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Abstract 


SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria  (MOSAIC)  is  a  management 
approach  for  establishing  and  maintaining  confidence  that  objectives  will  be  achieved  success¬ 
fully.  It  comprises  a  suite  of  risk-based  methods  for  assessing  and  managing  complex  projects  and 
processes.  The  Mission  Diagnostic  Protocol  (MDP)  is  one  of  the  assessments  included  in 
MOSAIC.  MDP  provides  a  time -efficient  means  of  analyzing  the  potential  for  success  in  complex 
and  uncertain  environments  and  can  be  applied  across  the  life  cycle  and  throughout  the  supply 
chain.  It  produces  a  broad  overview  of  the  current  state  of  risk  and  opportunity  for  a  project  or 
process.  With  MDP,  a  set  of  key  drivers  is  evaluated  to  establish  current  conditions  and  circum¬ 
stances  that  can  affect  performance.  Then,  a  simple  algorithm  is  used  to  estimate  the  likelihood  of 
achieving  the  objectives  being  pursued.  An  MDP  assessment  is  straightforward  to  conduct,  and  it 
can  be  self-applied  by  people  who  are  responsible  for  overseeing  projects  and  processes.  The  pur¬ 
pose  of  this  document  is  to  describe  the  core  set  of  activities  and  outputs  that  defines  MDP. 
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1  Introduction 


Mission  Diagnostic 
Protocol 


Purpose  of  this 
Document 


Intended  Audience 


SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria 
(MOSAIC)  is  a  management  approach  for  establishing  and  maintain¬ 
ing  confidence  that  objectives  will  be  achieved  successfully.  It  com¬ 
prises  a  suite  of  risk-based  methods  for  assessing  and  managing  com¬ 
plex  projects  and  processes.  The  Mission  Diagnostic  Protocol  (MDP) 
is  one  of  the  assessments  included  in  MOSAIC. 

MDP  is  a  risk-based  assessment  for  evaluating  current  conditions  and 
determining  whether  a  project  or  process  is  on  track  for  success.  MDP 
is  a  time -efficient  means  of  analyzing  the  potential  for  success  in  com¬ 
plex  and  uncertain  environments  and  can  be  applied  across  the  lifecy¬ 
cle  and  throughout  the  supply  chain.  An  MDP  assessment  is  straight¬ 
forward  to  conduct,  and  it  can  be  self-applied  by  people  who  are 
responsible  for  overseeing  projects  and  processes. 

An  MDP  assessment  provides  a  broad  overview  of  the  current  state  of 
risk  and  opportunity  for  a  project  or  process.  It  can  be  viewed  as  a 
first-pass  screening  to  diagnose  any  unusual  circumstances  that  might 
affect  the  potential  for  success.  More  detailed,  follow-on  evaluations 
might  be  required  when  the  potential  for  success  is  judged  to  be  unac¬ 
ceptable. 


The  purpose  of  this  document  is  to  define  the  core  set  of  activities  and 
outputs  that  defines  MDP  and  present  the  basic  approach,  or  frame¬ 
work,  for  conducting  an  MDP  assessment.  However,  this  document 
does  not  provide  step-by-step  procedures  for  performing  an  MDP  as¬ 
sessment.  Guidebook(s)  focusing  on  how  to  conduct  an  MDP  assess¬ 
ment  and  domain-specific  methods  consistent  with  MDP  will  be  pub¬ 
lished  in  the  future. 


The  primary  audience  for  this  technical  report  is  people  who  have  ex¬ 
perience  assessing  and  managing  risk  in  development  and  operational 
settings.  This  includes  people  who  oversee  complex  projects  and  proc¬ 
esses.  People  who  have  experience  with  or  are  interested  in  the  follow¬ 
ing  topics  might  also  find  this  document  useful: 

•  time-  and  resource-efficient  methods  for  assessing  and  managing 

risk 

•  general  project  or  program  management 

•  success-driven  management  of  projects  or  processes 
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Structure  of  this 
Document 


This  technical  report  is  divided  into  the  following  parts: 

•  Section  1:  Introduction — provides  a  brief  overview  of  MDP  and 

defines  the  audience  for  this  document 

•  Section  2:  MOSAIC — presents  background  information  about 

MOSAIC  and  its  assessment  methods 

•  Section  3:  Mission  Diagnostic  Protocol — describes  the  driver- 

based  approach  of  MDP,  including  an  overview  of  each  activity 

•  Section  4:  Summary  and  Future  Work — presents  a  brief  synopsis 

of  research  and  development  activities  related  to  MOSAIC  and 
MDP 

•  Appendix  A:  Risk  Management  Concepts — provides  a  basic  over¬ 

view  of  risk  management  concepts  and  philosophy 

•  Appendix  B:  Protocol  Structure  and  Nomenclature — describes  the 

standard  structure  and  naming  conventions  for  the  MDP  data  flows 
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2  MOSAIC 


Introduction 


A  New  Approach  for 
a  Complex  Problem 
Space 


Focus  on  Projects 
and  Processes 


This  section  provides  background  information  about  the  body  of  re¬ 
search  underlying  MOSAIC  and  MDP.  It  also  provides  significant 
concepts  and  terminology  needed  to  understand  MDP,  specifically: 

•  basic  structure  of  MOSAIC  assessment  methods 

•  focus  on  managing  key  objectives 

•  success-oriented  philosophy  of  MOSAIC 

•  driver-based  analysis  approach 


Today’s  business,  project,  and  operational  environments  are  becoming 
increasingly  complex.  People  often  struggle  to  make  sense  of  this 
complexity,  which  places  many  critical  projects  and  processes  at  risk 
of  failing.  MOSAIC  is  a  management  approach  that  establishes  and 
maintains  confidence  that  objectives  will  be  achieved  successfully.  It 
comprises  a  suite  of  risk-based  methods  for  assessing  and  managing 
complex  projects  and  processes  [Alberts  2007]. 

MOSAIC  is  a  highly  flexible  approach  that  can  be  applied  across  the 
life  cycle  and  used  to  manage  projects  and  processes  that  cross  organ¬ 
izational  boundaries.  It  is  designed  to  help  people  analyze  tradeoffs 
and  make  better  decisions  in  situations  that  have  a  high  degree  of  un¬ 
certainty.  MDP  is  one  of  the  assessments  included  in  MOSAIC. 


To  date,  MOSAIC  research  and  development  activities  have  primarily 
focused  on  assessing  the  success  potential  of  projects  and  processes. 
As  a  result,  this  document  examines  how  MDP  is  used  in  the  context 
of  projects  and  processes.  As  MDP  is  used  in  other  contexts  (e.g.,  to 
assess  technology),  additional  guidance  will  be  provided. 
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Projects 


Processes 


Outcome 

Management 


In  MOSAIC,  a  project  is  defined  as  a  set  of  activities  that  produces  a 
unique  product  for  a  customer  or  delivers  a  service  that  is  tailored  for  a 
customer’s  needs.  A  project  is  often  executed  only  once.  For  example, 
when  an  organization  develops  a  software -intensive  system  for  a  spe¬ 
cific  customer,  its  management  charters  a  project  to  develop  that  sys¬ 
tem.  The  project  begins  with  the  initial  concept  for  the  system  and 
ends  when  the  system  is  satisfactorily  delivered  to  the  customer.  Pro¬ 
jects  can  range  from  small  software  development  projects  with  5  or  10 
people  to  a  large  Department  of  Defense  (DoD)  systems  development 
program  that  includes  multiple  government  and  contractor  organiza¬ 
tions. 


In  contrast  to  a  project,  a  process  is  a  set  of  activities  that  is  typically 
executed  more  than  once.  Two  types  of  processes  are  considered  in 
MOSAIC:  business  and  operational  processes.  In  this  document,  a 
process  that  provides  a  core  business  function  is  called  a  business 
process.  For  a  healthcare  organization,  the  patient-care  workflow  is 
considered  to  be  a  core  business  process  because  it  directly  supports 
the  mission  of  the  organization  (i.e.,  to  provide  healthcare  services  to 
patients).  In  contrast,  an  operational  process  indirectly  supports  the 
mission  of  the  organization.  It  is  not  part  of  the  organization’s  reve¬ 
nue-producing  processes.  An  information  technology  (IT)  process  for 
configuring  and  maintaining  an  organization’s  computing  infrastruc¬ 
ture  is  an  example  of  an  operational  process.  The  term  process  as  used 
in  this  document  refers  to  both  operational  and  business  processes. 


MOSAIC  methods  help  decision  makers  establish  and  maintain  a  rea¬ 
sonable  degree  of  confidence  that  projects  and  processes  will  success¬ 
fully  achieve  their  defined  objectives.  The  overarching  goal  of  this  ap¬ 
proach  is  to  ensure  that  the  eventual  outcome,  or  result,  satisfactorily 
achieves  the  objectives  being  pursued.  The  focus  on  managing  out¬ 
comes  enables  decision  makers  to  balance  potential  gain  being  pursued 
(i.e.,  opportunity)  against  the  potential  losses  that  can  occur  (i.e.,  risk), 
which  then  enables  decision  makers  to  define  a  path  toward  achieving 
success. 
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Assessment 
Protocols, 
Activities,  and 
Techniques 


Protocol  Flexibility 


Supporting 

Artifacts 


Assessment 

Methods 


Each  MOSAIC  assessment  and  management  method  is  based  on  a 
specific  protocol.  As  used  in  this  context,  a  protocol  is  the  basic  ap¬ 
proach,  or  framework,  for  conducting  an  assessment  or  management 
method.  It  defines  the  sequence  of  activities  that  must  be  performed 
but  does  not  indicate  how  to  perform  those  activities.  You  can  think  of 
a  protocol  and  its  associated  activities  as  providing  the  basic  require¬ 
ments  for  conducting  an  assessment. 

A  technique  is  a  specific  practice  that  can  be  used  when  performing  a 
protocol  activity.  For  example,  consider  the  following  protocol  activ¬ 
ity:  Gather  data  from  people.  Many  interviewing  and  surveying  tech¬ 
niques  can  be  used  to  gather  data  from  people  who  are  knowledgeable 
about  a  subject.  The  objective  is  to  select  the  technique  that  is  most 
appropriate  for  your  circumstances.  In  some  cases,  an  interview  might 
be  the  best  choice,  while  in  other  instances  a  survey  that  people  com¬ 
plete  anonymously  would  be  more  appropriate.  Either  way,  you  get  the 
needed  information;  you  just  use  different  means  to  collect  it. 


While  you  can  use  a  single  technique  to  achieve  the  goals  of  a  given 
protocol  activity,  you  might  decide  to  combine  several  techniques  to 
meet  the  goals.  In  this  regard,  MOSAIC  offers  considerable  flexibility 
in  tailoring  an  assessment  to  a  particular  environment  or  set  of  circum¬ 
stances. 


When  you  conduct  any  technique,  you  will  likely  use  one  or  more  sup¬ 
porting  artifacts  to  gather,  analyze,  or  record  data.  Worksheets,  tem¬ 
plates,  and  tools  are  all  examples  of  supporting  artifacts.  Suppose  for 
the  protocol  activity  Gather  data  from  people  you  decide  to  conduct  an 
interview  with  a  set  of  carefully  chosen  participants.  During  the  inter¬ 
view  session,  you  frame  the  discussion  around  a  set  of  key  questions. 
That  list  of  questions,  which  is  essential  for  conducting  an  efficient  and 
effective  interview,  is  an  example  of  a  supporting  artifact. 


Protocols  (and  their  associated  activities),  techniques,  and  supporting 
artifacts  form  the  basis  for  assessment  methods  in  MOSAIC.  Figure  1 
shows  how  a  method  is  created  by  linking  techniques  and  supporting 
artifacts  with  a  protocol’s  activities.  The  collective  set  of  techniques 
and  artifacts  used  to  conduct  the  protocol  (represented  by  the  shaded 
boxes)  constitute  a  method  for  that  protocol. 
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Protocol 


Protocol  A 


Protocol  Activities 

Techniques 

Supporting  Artifacts 


Multiple  Methods 
Consistent  with  a 
Protocol 


Protocol 

Protocol  Activities 

Techniques 

Supporting  Artifacts 


Broad  Applicability 
of  MDP 


- - - 

Activity  A.  1  Activity  A.2  Activity  A.3 


Figure  1:  A  Method  Consistent  with  Protocol  A 

With  MOSAIC,  multiple  methods  can  be  consistent  with  a  given  pro¬ 
tocol,  as  illustrated  in  Figure  2.  A  common  protocol  forms  the  basis  for 
the  methods  illustrated  in  Figure  1  and  Figure  2.  Flowever,  the  two 
methods  incorporate  different  techniques  and  artifacts.  The  two  meth¬ 
ods  accomplish  the  same  objectives  as  defined  by  the  common  proto¬ 
col  they  follow,  but  each  incorporates  a  unique  combination  of  tech¬ 
niques  and  artifacts. 


Protocol  A 


>- 


Activity  A.1 


Activity  A.2  Activity  A.3 


Figure  2:  A  Second  Method  Consistent  with  Protocol  A 

The  protocol  defined  in  this  document,  MDP,  can  be  applied  to  many 
different  domains  and  types  of  problems.  To  date,  MDP  has  been  ap¬ 
plied  to  both  software  development  projects  and  operational  processes, 
which  illustrates  MDP’s  portability  across  the  life  cycle.  In  general,  the 
flexible  design  of  MOSAIC  assessment  and  management  methods  al¬ 
lows  them  to  be  applied  in  a  variety  of  domains  and  environments, 
across  the  life  cycle,  and  throughout  the  supply  chain.  The  main  focus 
when  applying  MDP  in  any  domain  or  problem  space  is  to  assess  the 
likelihood  that  key  objectives  will  be  achieved  successfully. 
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2.1  MISSIONS  AND  THEIR  OBJECTIVES 


What  is  a  Mission? 

The  term  mission  has  different  meanings,  depending  on  the  context  in 
which  it  is  used. "  For  example,  mission  is  used  to  describe  any  of  the 
following: 

•  the  purpose  of  an  organization 

•  the  goals  of  a  specific  department  or  group  within  a  larger  organi¬ 

zation 

•  the  specific  result  being  pursued  when  executing  a  project'  or  proc¬ 

ess 

•  the  objectives  of  each  activity  in  a  work  process 

•  the  function  of  each  technology  (e.g.,  a  software-intensive  system) 

that  supports  a  project  or  process 

Network  of 

Missions 

A  broad  network  of  missions  exists  within  all  organizations.  Success  at 
the  organizational  level  requires  ensuring  that  all  missions  within  the 
network  are  aligned.  Ensuring  alignment  among  an  organization’s 
missions  helps  establish  confidence  that  (1)  core  business  missions 
within  the  organization  will  be  achieved  and  (2)  the  organization’s 
overall  mission  will  also  be  accomplished. 

The  network  of  missions  can  also  extend  across  multiple  organizations. 
For  example,  when  multiple  companies  collaborate  on  a  joint  venture, 
such  as  building  and  fielding  a  complex  software-intensive  system, 
they  pool  their  resources  toward  achieving  a  common  mission.  Each 
organization  must  balance  its  local  objectives  against  the  shared  set  of 
objectives  defined  by  the  common  mission. 

MOSAIC  Definition 

of  Mission 

Within  the  context  of  projects  and  processes,  we  define  mission  as  the 
set  of  objectives,  or  desired  outcome,  of  a  project  or  process  within 
one  organization  or  spanning  multiple  organizations.  Put  another  way, 
the  mission  defines  what  success  looks  like  for  a  project  or  process. 

The  mission  of  a  project  or  process  typically  comprises  three  distinct 
types  of  objectives:  (1)  product  or  service,  (2)  cost,  and  (3)  schedule. 
These  three  objectives  define  the  tangible,  and  in  many  cases,  measur¬ 
able,  outcomes  being  pursued. 

2  We  assert  that  mission  is  a  recursive  concept. 

3  A  project,  as  defined  in  this  document,  includes  small,  independent  efforts  as  well  as  large  scale,  multi-organizational, 
geographically  distributed  DoD  programs. 
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Product  and 
Service  Objectives 


Cost  and  Schedule 
Objectives 


Picture  of  Success 


Product  objectives  define  the  nature  of  the  items  produced.  These  ob¬ 
jectives  are  often  referred  to  as  technical  objectives  in  the  software 
development  domain.  For  example,  if  you  are  developing  a  software¬ 
intensive  system,  the  product  (i.e.,  technical)  objectives  define  the 
functional  and  performance  characteristics  of  the  system  as  well  as 
other  desired  attributes,  like  safety  or  security.  Product  objectives  thus 
define  the  parameters  of  success  for  the  products  you  build. 

Service  objectives  define  the  nature  of  the  services  provided  to  the  re¬ 
cipients  of  those  services  (i.e.,  customers).  If  the  service  you  are  pro¬ 
viding  is  help-desk  support,  the  service  objectives  will  define  the  qual¬ 
ity  of  help-desk  support  provided  to  constituents  (such  as  the  required 
response  time  based  on  the  priority  of  the  request).  Service  objectives 
define  the  parameters  of  success  for  the  services  you  provide  to  cus¬ 
tomers. 


In  some  instances,  a  mission  is  defined  solely  by  its  product  or  service 
objectives.  However,  in  most  cases,  constraints  are  also  considered  in 
relation  to  product  or  service  objectives.  Managers  generally  do  not 
have  limitless  funds  at  their  disposal,  nor  do  they  have  an  infinite 
amount  of  time  in  which  to  complete  work  tasks.  As  a  result,  cost  and 
schedule  objectives  must  be  considered  alongside  product  or  service 
objectives  (and  in  many  cases  are  the  key  drivers  of  management’s 
decisions,  especially  as  time  goes  by  and  costs  accrue). 


Product  or  service,  cost,  and  schedule  objectives,  when  viewed  to¬ 
gether,  typically  define  the  basic  mission  of  a  project  or  process.  They 
specify  what  will  be  accomplished,  the  anticipated  costs  to  complete 
all  activities,  and  the  time  frame  in  which  work  will  be  completed. 
When  appropriate,  these  objectives  can  be  supplemented  with  other 
objectives  (such  as  business  or  financial  objectives)  to  produce  a  com¬ 
plete  picture  of  success.  The  mission,  or  picture  of  success,  defines  the 
desired  outcome  for  a  project  or  process.  Once  the  desired  outcome  is 
established,  management  activities  must  be  geared  toward  ensuring 
that  results  satisfy  that  outcome.  Risk  management  is  an  essential  part 
of  achieving  that  success.  (Appendix  A  of  this  document  highlights  the 
foundational  concepts  of  risk  management  as  used  in  MOSAIC.) 
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An  Incomplete 
Picture  Using 
Traditional  Risk 
Management 


Organizations  typically  manage  several  types  of  risk  using  traditional 
approaches,  including  project  risk,  security  risk,  technology  risk,  and 
operational  risk.  Each  type  of  risk  is  differentiated  by  the  unique 
sources,  or  causes,  that  produce  it.  Normally,  responsibility  for  manag¬ 
ing  different  types  of  risks  is  assigned  to  different  groups  within  an 
organization. 

Because  each  type  of  risk  is  normally  managed  in  isolation,  it  is  diffi¬ 
cult  to  establish  the  overall  success  potential  of  a  project  or  process 
using  traditional  risk-management  approaches.  Since  different  groups 
in  an  organization  have  responsibility  for  managing  different  types  of 
risk,  each  group  tends  to  locally  optimize  its  mitigation  efforts.  No  one 
is  responsible  for  consolidating  disparate  risk  data.  As  a  result,  the 
overall  chances  for  success  are  not  explicitly  determined.  In  contrast,  a 
MOSAIC  assessment  is  specifically  designed  to  establish  the  overall 
success  potential  of  a  project  or  process  by  analyzing  a  broad  range  of 
success  and  failure  drivers. 
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2.2  MANAGING  FOR  SUCCESS  USING  MOSAIC 


Success-Oriented 

Philosophy 


Potential  for 
Success 


Success  Profile 


The  MOSAIC  management  approach  requires  establishing  and  main¬ 
taining  a  reasonable  degree  of  confidence  that  project  or  process  ob¬ 
jectives  will  be  achieved  successfully.  This  success-oriented  philoso¬ 
phy  requires  managers  to  focus  their  attention  on  managing  the  result, 
or  outcome,  of  a  project  or  process.  The  goal  is  to  ensure  that  the  even¬ 
tual  outcome  fulfills  the  objectives  being  pursued. 

Traditional  risk-management  approaches  generate  a  set  of  risks  for  a 
project  or  process.  Each  risk  in  the  set  is  a  cause-effect  pair  that  con¬ 
veys  the  potential  consequence  triggered  by  a  single  condition  or 
event.  In  contrast,  MOSAIC 

•  Focuses  on  the  desired  outcome  (i.e..  the  objectives  being  pursued) 
and 

•  Examines  the  range  of  conditions  and  potential  events  that  influ¬ 
ence  the  chances  of  achieving  the  desired  outcome. 


The  potential  for  success  characterizes  the  likelihood,  or  probability, 
that  the  desired  outcome  will  be  achieved  or  exceeded.  It  can  be  ex¬ 
pressed  qualitatively  in  relation  to  a  set  of  evaluation  criteria  or  quanti¬ 
tatively,  depending  on  the  assessment  method  that  is  used. 


As  shown  in  Figure  3,  the  success  profile  depicts  the  current  potential 
for  success  in  relation  to  its  success  threshold,  which  is  the  desired,  or 
target,  potential  for  success.  The  success  threshold  separates  accept¬ 
able  values  of  the  potential  for  success  from  those  that  are  considered 
to  be  unacceptable. 

Excellent . 

High . 

Borderline . 

Low . 

Minimal . 

Current  potential  for  success 

Figure  3:  Success  Profile 


-  Success  threshold 

(desired  potential  for  success) 


> 


Success 

differential 
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Success 

Differential 

As  depicted  in  Figure  3,  the  success  differential  is  a  measure  of  the 
current  potential  for  success  in  relation  to  the  desired  value  as  defined 
by  the  success  threshold.  The  success  differential  illustrates  the  degree 
of  improvement  that  will  be  required  to  position  a  project  or  process 
for  success. 

Managing  the 

Potential  for 

Success 

When  applying  the  MOSAIC  approach,  people  (1)  assess  the  current 
potential  for  success  in  relation  to  its  desired  value  (i.e.,  its  success 
threshold)  and  (2)  take  planned  action,  when  appropriate,  to  bring  the 
potential  for  success  in  alignment  with  the  success  threshold. 

MOSAIC  requires  people  to  track  the  potential  for  success  over  time 
and  take  appropriate  action  as  needed  to  ensure  that  the  potential  for 
success  is  kept  within  an  acceptable  tolerance. 

Mission  Assurance 

Mission  assurance  is  defined  as  the  level  of  confidence  that  mission 
objectives  will  be  achieved  successfully.  When  viewed  within  the  con¬ 
text  of  a  project  or  a  process,  mission  assurance  focuses  on  establish¬ 
ing  and  maintaining  an  appropriate  level  of  confidence  in  the  potential 
for  achieving  project  or  process  objectives.  MOSAIC  is  a  means  of 
achieving  the  desired  level  of  mission  assurance  for  projects  or  proc¬ 
esses. 
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2.3  ANALYZING  KEY  DRIVERS  OF  SUCCESS 


Purpose  of 
Research  and 
Development 


What  is  a  Driver? 


Success  and  Failure 
Drivers 


An  extensive  and  time-consuming  analysis  is  normally  required  when 
conducting  most  commonly  used  risk  assessments.  An  underlying  goal 
of  the  Software  Engineering  Institute  (SEI)  research  and  development 
activities  related  to  MDP  is  to  demonstrate  that  an  assessment  does  not 
require  a  significant  investment  of  time  to  be  effective.  In  this  way, 
people  can  more  readily  adopt  risk-based  approaches  that  help  them 
weigh  the  alternatives  confronting  them  and  ultimately  make  better 
decisions. 

The  goal  of  all  MOSAIC  assessments,  including  MDP  assessments,  is 
to  determine  the  success  potential  of  a  project  or  process.  This  focus  on 
managing  success  clearly  distinguishes  MOSAIC  from  traditional  risk 
management,  in  which  the  goal  is  to  avoid  failure.  A  key  aspect  of 
MOSAIC’S  success-oriented  approach  is  being  able  to  assess  a  pro¬ 
ject’s  or  process’  overall  chances  of  succeeding.  An  MDP  assessment 
determines  the  potential  for  success  by  using  a  simple  algorithm  to 
analyze  a  small  set  of  key  outcome  drivers. 


Drivers  are  characteristics  of  a  project  or  process  that  are  essential  for 
achieving  its  objectives.  Each  individual  driver  has  a  strong  influence 
on  the  ultimate  outcome,  or  result.  The  cumulative  effects  of  all  drivers 
can  be  analyzed  to  determine  whether  a  project  or  process  has  suffi¬ 
cient  momentum  toward  its  objectives.  Establishing  the  effects  of  driv¬ 
ers  is  crucial  when  analyzing  the  potential  for  success. 


In  MDP  assessments,  each  driver  is,  by  definition,  worded  as  a  success 
driver.  Consider  the  following  example:  Task  execution  is  effective  and 
efficient.  Here,  the  implication  is  that  people  have  sufficient  capability 
to  complete  their  assigned  tasks.  This  is  obviously  a  positive  character¬ 
istic  of  a  project  or  process  that  helps  enhance  its  potential  for  success, 
which  makes  it  a  success  driver. 

Further,  MDP  assessments  determine  which  drivers  from  a  set  are 
guiding  a  project  toward  a  successful  outcome  and  which  are  not. 

When  a  given  driver  does  not  have  a  positive  influence  on  a  project  or 
process,  it  is  acting  as  a  failure  driver.  Here,  the  driver  is  reducing  the 
momentum  toward  achieving  objectives  and  making  an  unsuccessful, 
or  failed,  outcome  more  likely.  For  example,  if  people  do  not  have  suf¬ 
ficient  capability  to  complete  their  assigned  tasks,  the  success  potential 
of  the  project  or  process  is  adversely  affected. 
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DRIVER  Questions  Each  driver  is  characterized  as  a  yes/no  question,  where  an  answer  of 

yes  denotes  a  success  driver  and  an  answer  of  no  denotes  a  failure  driv¬ 
er.  Examples  of  driver  questions  used  in  an  MDP  assessment  include 

•  Are  project  goals  realistic  and  well  articulated? 

•  Are  customer  requirements  and  needs  well  understood? 

•  Are  organizational  and  political  conditions  facilitating  completion 

of  project  activities? 

Note:  Refer  to  Example  1 :  Set  of  Drivers  on  page  28. 

Since  an  important  aspect  of  an  MDP  assessment  is  time  efficiency, 
you  need  to  keep  the  number  of  drivers  small  to  ensure  that  the  as¬ 
sessment  can  be  completed  in  a  reasonable  amount  of  time.  Experience 
has  shown  that  good  results  are  achieved  by  using  between  10  and  15 
drivers  in  an  assessment. 


Tailoring  Drivers  to 
Reflect  Key 
Characteristics  of 
Success 


Whenever  you  tailor  drivers  for  an  assessment,  you  need  to  make  sure 
that  the  driver  set  addresses  all  key  aspects  of  the  project  or  process 
being  assessed.  In  general,  you  should  ensure  that  a  set  of  drivers  mi¬ 
nimally  addresses  the  following  aspects  of  a  project  or  process: 

•  the  project  or  process  objectives,  including  technical,  funding  and 
schedule  objectives 

•  the  product  being  developed  or  the  service  being  provided 

•  planning  and  preparing  to  execute  a  project  or  process 

•  execution  of  tasks  and  activities 

•  the  operational  and  business  environments 

•  capacity  and  capability  to  manage  unpredictable,  external  events 


The  driver  set  should  be  tailored  for  each  specific  context  because  it  is 
essential  that  drivers  provide  meaningful  information  about  a  project  or 
process.  A  generic  set  of  drivers,  such  as  those  featured  in  Example  1: 
Set  of  Drivers  on  page  28,  can  be  used  as  a  starting  point  for  an  as¬ 
sessment.  However,  you  need  to  ensure  that  the  driver  set  used  to  as¬ 
sess  a  project  or  process  accurately  reflects  the  key  characteristics  that 
define  success  for  that  project  or  process. 
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Basic  Analysis 
Approach 


MDP  Analysis  Based 
on  Algorithms 


The  philosophy  underlying  MDP  is  that  the  relative  impact  of  drivers 
can  be  used  to  gauge  the  potential  for  success.  A  predominance  of  suc¬ 
cess  drivers  in  relation  to  failure  drivers  indicates  an  acceptable  poten¬ 
tial  for  success  (and  vice  versa).  After  each  driver  is  evaluated  to  de¬ 
termine  its  effect  on  the  outcome,  a  simple  algorithm  is  used  to 
estimate  the  potential  for  success  based  on  the  aggregate  effect  of  all 
drivers. 

The  analysis  of  drivers  in  MDP  can  be  conceptually  broken  into  the 
following  two  parts: 

1 .  Evaluate  each  driver  to  determine  the  extent  to  which  each  is  pre¬ 
sent.  (See  Section  3.3.3,  Evaluate  Drivers  (Activity  A3),  for  more 
details.) 

2.  Analyze  the  entire  set  of  drivers  to  determine  the  potential  for 
success.  (See  Section  3.3 .4,  Apply  Analysis  Algorithm  (Activity 
A4),  for  more  details.) 


Each  MOSAIC  assessment  protocol  is  based  on  an  analysis  approach 
that  uniquely  defines  that  protocol.  The  unique  aspect  of  MDP  is  its 
algorithm-based  analysis.  An  algorithm,  as  defined  in  this  document,  is 
defined  as  a  finite  list  of  well-defined  instructions  for  accomplishing  a 
given  task  that,  given  an  initial  state,  will  produce  a  defined  end-state, 
or  result. 

An  MDP  assessment  employs  an  algorithm  that  uses  simple  mathemat¬ 
ics  or  rule-based  logic  to  analyze  a  set  of  drivers.  The  objective  is  to 
use  a  simple  set  of  instructions  to  estimate  the  potential  for  success 
using  a  predefined  set  of  criteria  (called  success  criteria).  Several  types 
of  algorithms  that  can  be  used  are  discussed  further  in  Section  3.3.4, 
Apply  Analysis  Algorithm  (Activity  A4 ). 
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3  Mission  Diagnostic  Protocol 


Introduction 

This  section  describes  MDP.  It  begins  with  an  overview  of  MDP  proc¬ 
esses  and  activities.  Then,  details  for  key  activities  are  provided  along 
with  selected  examples.  The  examples  are  not  meant  to  be  all- 
inclusive;  rather  they  are  provided  to  assist  the  reader  in  understanding 
what  an  activity  accomplishes.  Other  documents  that  expand  on  this 
material  and  provide  specific  step-by-step  directions  for  different  ac¬ 
tivities  and  techniques  (e.g.,  guidebooks,  method  descriptions)  are 
planned  for  future  publication. 

Purpose 

An  MDP  assessment  provides  a  first-pass  screening  of  a  project  or 
process  to  identify  any  major  issues  or  circumstances  that  can  affect  its 
potential  for  success.  It  can  be  self-applied4  by  people  who  have  ex¬ 
perience  and  expertise  working  in  the  domain  area  being  assessed. 

Objectives 

The  main  objectives  of  an  MDP  assessment  are  to 

•  assess  the  potential  for  success  by  evaluating  a  small  set  of  drivers 
in  relation  to  current  conditions 

•  determine  whether  the  current  potential  for  success  is  acceptable 

•  identify  actions  to  maintain  or  improve  the  current  potential  for 

success 

Assessment 

Benefits 

When  used  properly,  an  MDP  assessment  provides  an  effective  diag¬ 
nosis  of  the  major  issues  affecting  a  project’s  or  process’  potential  for 

success. 

An  MDP  assessment  provides  a  simple  means  of  gauging  the  potential 
for  success.  It  is  a  time-  and  resource-efficient  way  to  identify  major 
issues  that  can  affect  a  project  or  process. 

People  do  not  need  to  be  experts  in  applying  and  tailoring  MDP  as¬ 
sessments  to  obtain  actionable  assessment  results. 

4  An  MDP  assessment  can  be  self-applied  by  the  team  responsible  for  overseeing  or  executing  a  project  or  process.  Alter¬ 
natively,  it  can  be  applied  by  a  third  party  on  behalf  of  the  team  responsible  for  executing  a  project  or  process.  In  either 
case,  the  core  MDP  activities  performed  are  identical.  However,  the  techniques  and  artifacts  used  to  conduct  the  as¬ 
sessment  might  vary. 
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Assessment 

Limitations 


Tailoring  an  MDP  assessment  (see  Phase  1  on  page  18  for  more  infor¬ 
mation)  for  a  specific  domain  or  problem  space  requires  some  experi¬ 
ence  and  expertise.  Some  issues  can  elude  detection  when  people  use  a 
generic  set  of  drivers  rather  than  a  set  that  uniquely  reflects  the  specific 
project  or  process  being  assessed. 


An  MDP  assessment  only  examines  the  extent  to  which  conditions  are 
favorable  for  a  successful  outcome  using  a  basic  analysis  approach.  An 
MDP  assessment  only  provides  a  “ballpark”  measure  of  the  potential 
for  success. 


SKILLS  Required  MDP  is  normally  performed  by  a  small  team  (sometimes  referred  to  as 

an  analysis  team)  with  the  following  skills5: 

•  detailed  knowledge  of  the  domain  in  which  the  project  or  process  is 
executed 

•  knowledge  of  process  improvement  and  management 

•  knowledge  and  skills  appropriate  to  applying  MDP,  such  as 

-  analytical  skills 

-  interviewing  skills 

-  facilitation  skills 

-  note-taking  skills  (i.e.,  ability  to  quickly  record  data  that  are 
identified  by  participants) 

-  communication  skills 


5  These  skills  can  be  distributed  across  a  number  of  people  in  a  team.  Some  people  may  have  multiple  skills  and  others 
may  be  specialists.  What  is  important  is  that  the  team  performing  the  MDP,  as  a  whole,  has  this  set  of  skills. 
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3.1  PROTOCOL  STRUCTURE 


PHASED  Assessment  The  goal  of  each  MOSAIC  assessment  protocol  is  to  specify  a  se- 
APPROACH  quence  of  activities  that  must  be  performed  when  conducting  that  as¬ 

sessment.  However,  an  assessment  must  be  performed  within  a  broader 
context,  or  environment.  Therefore,  the  protocol  structure  used  within 
MOSAIC  also  specifies  preparation  and  post-assessment  activities. 
Figure  4  shows  the  three  phases  that  must  be  completed  when  conduct¬ 
ing  MOSAIC  assessments. 


Phase  1 

\  Prepare  for  the 
/  Assessment 


Phase  2 
Conduct  the 
Assessment 


Phase  3 
Complete  Post- 
Assessment  Activities 


Figure  4:  Protocol  Structure 

PROTOCOL  The  focal  point  of  a  MOSAIC  assessment  protocol  is  a  dataflow  dia- 

DATAFLOWS  gram.  For  each  assessment  protocol,  the  following  diagrams  are  docu¬ 

mented: 

•  a  high-level  dataflow  diagram  for  each  phase 

•  a  detailed  dataflow  diagram  for  Phase  2 

•  a  high-level  dataflow  diagram  for  each  Phase  2  activity 

Phase  2  is  described  in  more  detail  than  the  other  two  phases  because  it 
specifies  the  distinct  sequence  of  activities  that  uniquely  defines  the 
assessment  approach.  In  other  words,  the  unique  characteristics  of  the 
assessment  are  embodied  in  its  Phase  2  activity  dataflow.  The  prepara¬ 
tion  and  post-assessment  activities  of  Phases  1  and  3  are  common  to  all 
assessment  protocols  and  do  not  have  a  unique  sequence  of  activities 
associated  with  them.  Only  a  top-level  dataflow  is  presented  for  Phases 
1  and  3.  More  detailed  information  about  the  structure  of  MOSAIC 
assessment  protocols  is  presented  in  Appendix  B:  Protocol  Structure 
and  Nomenclature. 
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3.2  PREPARE  FOR  THE  ASSESSMENT  (PHASE  1) 


INTRODUCTION  Phase  1  of  an  MDP  assessment,  Prepare  for  the  Assessment,  is  focused 

on  getting  ready  to  conduct  the  assessment.  This  includes  all  of  the 
planning  and  logistics  management  needed  to  make  the  assessment 
execution  flow  smoothly  as  well  as  assuring  that  key  stakeholders  pro¬ 
vide  visible  support  for  the  assessment.  This  preparation  lays  the  foun¬ 
dation  for  conducting  the  assessment  during  Phase  2. 


OBJECTIVES  Phase  1  answers  the  following  questions: 

•  Who  is  sponsoring  the  assessment? 

•  How  can  stakeholder  sponsorship  be  attained? 

•  What  is  the  scope  of  the  assessment? 

•  What  is  the  plan  for  conducting  the  assessment? 

•  How  will  the  assessment  team  gain  the  knowledge,  skills,  and  abili¬ 
ties  to  perform  the  assessment? 

•  What  facilities  and  equipment  are  needed  to  conduct  each  assess¬ 
ment  activity? 

•  What  procedures,  tools,  and  artifacts  are  needed  to  conduct  each 
assessment  activity? 


DATAFLOW  The  following  diagram  highlights  the  data  flow  for  this  protocol  phase. 


Constraint 

Cl  Assessment  constraints 


R2  MDP  preparation  procedures 
R3  MDP  preparation  artifacts  &  tools 
R4  MDP  assessment  training  artifacts 
R5  Experienced  personnel 


Outputs 

PROI  Stakeholder  sponsorship 

PR02  Assessment  scope 

PR03  Assessment  plan 

PR04  Assessment  logistics 

PR05  Trained  personnel 

PR06  MDP  assessment  procedures 

PR07  MDP  assessment  artifacts  &  tools 


Figure  5:  Dataflow  for  MDP  Phase  1 
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Input 


The  following  input  is  required  by  the  activities  performed  during  this 
protocol  phase. 


Type 

Description 

PRI1  Assessment  require¬ 
ments 

The  goals  of  the  assessment,  needs  of  the  stakeholders,  and  a  basic  descrip¬ 
tion  of  the  project  or  process  being  analyzed 

CONSTRAINT  The  following  constraint  affects  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

Cl  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

RESOURCES  The  following  resources  support  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

R1  Mission  Diagnostic 
Protocol  (MDP)6 

The  basic  approach,  or  framework,  for  conducting  an  MDP  assessment 

R2  MDP  preparation  pro¬ 
cedures 

Documentation  that  describes  how  to  prepare  for  an  MDP  assessment 

R3  MDP  preparation  arti¬ 
facts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  prepare  for  an  MDP 
assessment 

R4  Assessment  training 
artifacts 

Documentation  and  other  materials  used  to  train  people  how  to  conduct  an 

MDP  assessment 

R5  Experienced  personnel 

People  who  are  experienced  in  all  phases  of  an  MDP  assessment 

OUTPUTS  The  following  outputs  are  produced  by  the  activities  performed  during 

this  protocol  phase. 


Type 

Description 

PROI  Stakeholder  spon¬ 
sorship 

Active  and  visible  support  of  the  assessment  by  key  stakeholders  and  decision 
makers 

6  Note  that  an  existing  method  consistent  with  the  protocol  will  include  all  of  the  procedures,  artifacts,  and  tools  required  to 
perform  the  assessment.  For  this  protocol,  it  is  assumed  a  method  is  created  as  part  of  preparation.  If  a  method  already 
exists  that  is  appropriate,  then  it  would  take  the  place  of  resources  R1 ,  R2,  and  R3. 
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Type 

Description 

PR02  Assessment  scope7 8 

The  boundaries  of  the  assessment,  including 

•  each  key  objective  for  the  project  or  process 

•  all  activities  needed  to  achieve  the  key  objectives 

•  the  people  who  have  ultimate  responsibility  for  completing  or  overseeing 
each  project  or  process  activity 

PR03  Assessment  plan 

The  approach  for  conducting  the  assessment,  including  key  activities,  re¬ 
sources,  schedule,  and  funding,  as  well  as  the  requirements  for  communicating 
results  to  key  stakeholders  after  the  assessment  is  complete 

PR04  Assessment  logistics 

The  facilities  and  equipment  needed  to  conduct  the  assessment  as  well  as 
communications  about  meeting  times  and  locations 

PR05  Trained  personnel 

The  people  who  are  tasked  with  performing  the  assessment  and  are  prepared 
to  conduct  it 

PR06  MDP  assessment 
procedures 

Documentation  that  describes  how  to  conduct  assessment  activities 

PR07  MDP  assessment 

artifacts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  perform  assessment 
activities 

Key  ACTIVITIES  The  following  table  highlights  the  activities  performed  during  this  pro- 

g 

tocol  phase. 


Activity 

Description 

Develop  stakeholder  spon¬ 
sorship 

Meet  with  key  stakeholders  and  decision  makers  to  foster  their  active  and  visi¬ 
ble  support  of  the  assessment 

Set  the  scope  of  the  as¬ 
sessment 

Determine  the  boundaries  of  the  assessment  based  on  requirements  and  con¬ 
straints  (schedule,  funding,  logistics,  contractual  restrictions) 

Develop  the  assessment 
plan 

Create  a  plan  for  conducting  the  assessment  based  on  its  scope  as  well  as 
requirements  and  constraints  (schedule,  funding,  etc.). 

Coordinate  logistics 

Reserve  rooms  for  meetings,  make  sure  that  any  required  equipment  (e.g., 
overhead  projectors,  flip  charts)  is  available,  and  inform  people  when  meetings 
will  be  held 

Train  personnel 

Ensure  that  people  who  will  perform  the  assessment  are  able  to  effectively 
conduct  all  assessment  activities 

Tailor  assessment  proce¬ 
dures,  criteria,  and  support¬ 
ing  artifacts9 

Adapt  all  MDP  assessment  procedures,  criteria,  and  supporting  artifacts  (e.g., 
worksheets,  templates,  tools)  for  the  circumstances  and  contexts  in  which 
those  procedures  will  be  used 

7  The  scope  defines  which  activities  to  include  in  the  assessment  and  becomes  a  constraint  in  Phase  2.  Some  aspects  of  a 
project  or  process  might  be  excluded  from  an  assessment  due  to  contract  limitations  or  on  the  basis  of  cost. 

8  Detailed  descriptions  of  Phase  1  activities  are  not  provided  in  this  document. 

9  The  set  of  drivers  is  considered  to  be  an  assessment  artifact.  Tailoring  the  set  of  drivers  for  a  given  application  of  MDP  is 
completed  during  Phase  1. 
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MDP  Tailoring 
Considerations 


An  MDP  assessment  must  be  tailored  for  the  context  in  which  it  is  ap¬ 
plied.  The  table  below  highlights  some  areas  in  which  an  MDP  as¬ 
sessment  is  commonly  tailored. 


Item 

Description 

Techniques 

The  specific  practices  used  to  perform  protocol  activities 

Selected  techniques  must  satisfactorily  achieve  the  key  outcomes  of  the  as¬ 
sessment  protocol  being  implemented. 

Procedures 

The  steps  followed  when  performing  a  technique 

Procedures  for  implementing  a  given  technique  must  be  consistent  with  the 
objectives  and  requirements  of  that  technique.  They  must  also  address  any 
constraints  and  unique  circumstances  encountered  (e.g.,  modifying  an  inter¬ 
view  technique  for  use  during  a  teleconference  rather  than  a  face-to-face  inter¬ 
view). 

Driver  set 

The  characteristics  of  a  project  or  process  essential  for  achieving  its  objectives 

The  cumulative  effects  of  all  drivers  are  analyzed  to  determine  whether  a  pro¬ 
ject  or  process  has  sufficient  momentum  toward  its  objectives.  The  driver  set 
used  to  assess  a  project  or  process  must  be  tailored  to  accurately  reflect  the 
key  success  characteristics  of  that  project  or  process. 

Assessment  criteria 

A  set  of  measures  used  in  various  aspects  of  the  assessment 

An  MDP  assessment  requires  the  following  criteria: 

•  Driver  value  criteria  in  Activity  A3  to  evaluate  each  individual  driver 

•  Success  criteria  in  Activity  A4  to  determine  the  potential  for  success 

All  criteria  used  during  an  MDP  assessment  must  reflect  the  requirements  and 
needs  of  key  decision  makers  and  stakeholders. 

Supporting  artifacts 

Worksheets,  templates,  and  tools  used  to  support  the  execution  of  a  given 
technique 

All  supporting  artifacts  must 

•  be  consistent  with  the  given  techniques  being  used 

•  support  the  key  outcomes  of  the  assessment  protocol  being  implemented 

•  support  the  overall  goals  of  the  assessment 
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3.3  CONDUCT  THE  ASSESSMENT  (PHASE  2) 


INTRODUCTION  During  Phase  2,  the  core  assessment  activities  are  performed.  During 

this  phase,  data  are  gathered  from  people  and  generated  from  relevant 
documentation.  These  data  are  then  used  to  evaluate  a  set  of  key  driv¬ 
ers  and  ultimately  determine  the  current  potential  for  success.  Deci¬ 
sion-makers  then  determine  whether  the  current  state  is  acceptable  and 
identify  actions  for  maintaining  or  improving  the  current  potential  for 
success. 


OBJECTIVES  This  protocol  phase  answers  the  following  questions: 

•  What  is  the  current  potential  for  success? 

•  Is  the  current  potential  for  success  acceptable? 

•  How  can  the  potential  for  success  be  maintained  or  improved  over 
time? 

DATAFLOW  The  following  diagram  highlights  the  dataflow  for  this  protocol  phase. 


Constraints 

Cl  Assessment  constraints 
PROI  Stakeholder  sponsorship 
PR02  Assessment  scope 
PR03  Assessment  plan 
PR04  Assessment  logistics 


Inputs 

11  People’s  knowledge 

12  Documentation 


▼ 

Outputs 

01  Driver  values  &  rationale 
>  02  Current  potential  for  success 
03  Success  profile 
04  Next  steps 


Phase  2 
Conduct  the 
assessment 


Resources 

PR05  Trained  personnel 

PR06  MDP  assessment  procedures 

PR07  MDP  assessment  artifacts  &  tools 


Figure  6:  Dataflow  for  MDP  Phase  2 


Inputs 


The  following  inputs  are  required  by  the  activities  performed  during 
this  protocol  phase. 
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Type 

Description 

11  People’s  knowledge 

People’s  individual  and  collective  perspectives,  information,  and  opinions  about 
the  project  or  process  and  its  potential  for  success 

12  Documentation 

Documentation  that  is  relevant  to  the  project  or  process.  Examples  include 
mission  statement,  policies,  procedures,  process  workflow,  work  products,  and 
quality  assurance  data. 

Constraints10 


The  following  constraints  affect  execution  of  the  activities  performed 
during  this  protocol  phase. 


Type 

Description 

Cl  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

PROI  Stakeholder  spon¬ 
sorship 

Active  and  visible  support  of  the  assessment  by  key  stakeholders  and  decision 
makers 

PR02  Assessment  scope 

The  boundaries  of  the  assessment,  including 

•  each  key  objective  for  the  project  or  process 

•  all  activities  needed  to  achieve  the  key  objectives 

•  the  people  who  have  ultimate  responsibility  for  completing  or  overseeing 
each  project  or  process  activity 

PR03  Assessment  plan 

The  activities  needed  to  conduct  the  assessment,  including  resources  and 
schedule  needed  to  complete  the  assessment 

PR04  Assessment  logistics 

The  facilities  and  equipment  needed  to  conduct  the  assessment  as  well  as 
communications  about  meeting  times  and  locations 

Resources10 


The  following  resources  support  execution  of  the  activities  performed 
during  this  protocol  phase. 


Type 

Description 

PR05  Trained  personnel 

People  who  understand  how  to  conduct  the  MDP  assessment 

PR06  MDP  assessment 
procedures 

Documentation  that  describes  how  to  conduct  each  technique  associated  with 
the  assessment  activity 

PR07  MDP  assessment 

artifacts  and  tools 

Worksheets,  automated  tools,  and  databases  needed  to  perform  the  tech¬ 
niques  associated  with  each  assessment  activities 

OUTPUTS  The  following  outputs  are  produced  by  the  activities  performed  during 

this  protocol  phase. 


10  Constraints  affect  all  activities  performed  during  Phase  2,  while  resources  are  used  to  aid  the  completion  of  all  activities 
performed  during  Phase  2.  The  definitions  for  all  Phase  2  constraints  and  resources  are  provided  in  this  section  only;  they 
are  not  provided  in  the  sections  for  individual  Phase  2  activities. 
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Type 

Description 

01  Driver  values  and  ra¬ 
tionale 

The  current  status  of  each  driver,  which  includes 

•  the  driver  value 

•  rationale  that  explains  why  that  value  was  selected 

02  Current  potential  for 

success 

The  current  probability,  or  likelihood,  that  the  desired  outcome  will  be  achieved 
or  exceeded 

03  Success  profile 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  tor  success,  or  success  threshold 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold 

04  Next  steps 

Actions  for  maintaining  or  improving  the  current  potential  for  success 

Key  ACTIVITIES  The  following  table  highlights  the  activities  performed  during  this  pro¬ 

tocol  phase.  The  remainder  of  this  section  provides  additional  details 
about  the  activities  featured  in  the  dataflow. 


Activity 

Description 

A1  Gather  data  from  people 

Elicit  information  about  a  project  or  process  from  people  who  play  a  role  in 
executing  it  and  transform  the  information  into  usable  data 

A2  Generate  data  from 

documentation 

Collect  documentation  relevant  to  the  project  or  process  (policies,  procedures, 
or  reports)  and  generate  usable  data  from  that  documentation 

A3  Evaluate  drivers 

Establish  the  current  conditions  affecting  the  project  or  process  by  evaluating 
the  key  drivers  of  success 

A4  Apply  analysis  algorithm 

Follow  the  selected  analysis  algorithm  to  estimate  the  current  potential  for  suc¬ 
cess 

A5  Establish  success  pro¬ 
file 

Generate  a  success  profile  for  the  project  or  process  by 

•  setting  the  success  threshold 

•  comparing  the  current  potential  for  success  to  the  success  threshold 

•  deciding  whether  or  not  the  current  potential  for  success  is  acceptable 

A6  Determine  next  steps 

Identify  actions  for  maintaining  or  improving  the  potential  for  success 

DETAILED  Dataflow  Figure  7  provides  a  detailed  dataflow  for  MDP  Phase  2. 
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Cl  Assessment  constraints 
PR01  Stakeholder  sponsorship 
PR02  Assessment  scope 
PR03  Assessment  plan 
PR04  Assessment  logistics 


■S 

c 

5 


z 

o 


.  CD 
CO 


!  C 
i  CO 


o  O 

■g* 

o  <“  2 
|  £$ 
e  £■'! 

O  C 
T3  £  0) 

’  g  $  E 

<0  Cfl  « 
£  c/>  a> 
I—  CO  U5 

8* 

O' 

0. 


<N 

Q 


a, 

a. 


£ 

q 

Q 


^3 


a 


K 

Go 

£ 


SOFTWARE  ENGINEERING  INSTITUTE  |  25 


3.3.1  GATHER  DATA  FROM  PEOPLE  (ACTIVITY  A1 ) 


INTRODUCTION  In  order  to  analyze  the  potential  for  success,  you  must  first  gather  rele¬ 

vant  information.  One  key  source  of  information  is  the  people  who 
perform  the  activities  that  support  the  project  or  process  (e.g.,  software 
developers  creating  a  product,  network  experts  responding  to  security 
incidents).  This  protocol  activity  (1)  elicits  information  from  selected 
personnel  who  play  a  role  in  executing  a  project  or  process  and  (2) 
transforms  their  tacit  knowledge  into  usable  data.  Information  that  is 
gathered  from  people  must  support  the  analysis  and  be  in  a  form  that  is 
compatible  with  chosen  techniques. 

OBJECTIVES  This  activity  answers  the  following  questions: 

•  What  conditions  and  events  are  driving  the  project  or  process  to¬ 
ward  a  successful  outcome? 

•  What  conditions  and  events  are  driving  the  project  or  process  to¬ 
ward  an  unsuccessful,  or  failed,  outcome? 


3.3.1. 1  Dataflow 


Constraints 


v 


Output 

>  N1  Data  from  people 


A 

Resources 

Figure  8:  Inputs  and  Outputs  for  Activity  A1 


Input 

II  People’s  knowledge 


A1 

Gather  data 
from  people 


Input  and  Output 

Description 

11  People’s  knowledge 

People’s  individual  and  collective  perspectives,  information,  and  opinions  about 
the  project  or  process  and  its  potential  for  success 

N1  Data  from  people 

Usable  data  about  a  project  or  process  based  on  individual  and  group  perspec¬ 
tives,  information,  and  opinions  about  the  project  or  process  and  its  potential  for 

success 
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3.3. 1.2  Techniques 


Techniques 

The  techniques  chosen  for  this  protocol  activity  depend  upon  the  na¬ 
ture  of  the  project  or  process  being  assessed,  the  knowledge,  skills,  and 
abilities  of  the  people  who  are  performing  the  assessment,  as  well  as 
organizational  practices,  culture,  and  constraints.  Several  techniques 
can  be  employed  to  collect  data  from  people.  Some  of  the  more  com¬ 
mon  data  collection  techniques  include  workshops,  interviews,  and 
surveys.  Each  is  briefly  described  below. 

Workshops 

A  workshop  is  a  facilitated  session  that  is  usually  focused  on  solving 
one  or  more  issues  or  problems.  The  facilitator(s)  and  participants 
work  together  when  investigating  the  issues  or  problems  in  a  work¬ 
shop.  Workshops  tend  to  foster  a  collaborative  environment. 

Interviews 

An  interview  is  a  facilitated  session  where  participants  answer  a  series 
of  specific  questions  asked  by  one  or  more  interviewers.  An  interview 
tends  to  be  more  formal  than  a  workshop  and  is  normally  focused  on 
data  elicitation  rather  than  problem  solving. 

Surveys 

A  survey  is  a  time -effective  way  to  gather  data  from  a  large  group  of 
people.  Respondents  are  provided  with  a  list  of  written  questions  and  a 
set  of  instructions  for  completing  the  survey.  In  most  cases,  people 
responding  to  a  survey  have  little  interaction  with  those  who  are  col¬ 
lecting  the  information,  making  surveys  an  impersonal  means  of  col¬ 
lecting  data.  During  an  interview  session,  a  survey  can  be  combined 
with  a  follow-on  discussion  to  stimulate  discussion,  clarify  survey  an¬ 
swers,  identify  conflicts,  and  elicit  issues  not  captured  by  the  survey. 
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PRELIMINARY  Data  A  considerable  amount  of  data  can  be  collected  during  workshops  and 

ANALYSIS  interviews;  not  all  of  that  data  will  be  relevant  to  analyses  that  will  be 

performed  later.  Sometimes  it  is  prudent  to  perform  a  preliminary 
analysis  of  data  to  eliminate  extraneous  data.  Subsequent  analyses  can 
often  be  conducted  more  efficiently  and  effectively  when  extraneous 
data  have  been  removed  from  the  input  data  set.  Preliminary  data  anal¬ 
ysis  is  an  optional,  first-pass  analysis  to  reduce  the  amount  of  data, 
clarify  uncertainty,  eliminate  non-useful  data,  and  identify  missing  or 
inadequate  data. 

3.3. 1.3  Examples 


EXAMPLE  Driver  Set  During  Phase  1,  a  set  of  drivers  is  tailored  for  the  project  or  process 

that  is  being  assessed.  In  an  MDP  assessment,  each  driver  is  character¬ 
ized  by  a  yes/no  question,  where  an  answer  of  yes  denotes  a  success 
driver  and  an  answer  of  no  denotes  a  failure  driver.  Example  1  below 
illustrates  a  set  of  driver  questions.  The  driver  questions  can  be  embod¬ 
ied  in  surveys  or  used  as  interview  questions  to  support  data  gathering 
efforts. 


Driver  Questions 

1 .  Are  project  goals  realistic  and  well-articulated? 

2.  Are  communication  and  information  sharing  about  project  activities  effective? 

3.  Are  customer  requirements  and  needs  well  understood? 

4.  Are  organizational  and  political  conditions  facilitating  completion  of  project  activities? 

5.  Is  the  project  plan  sufficient? 

6.  Does  project  management  facilitate  execution  of  tasks  and  activities? 

7.  Is  task  execution  efficient  and  effective? 

8.  Is  staffing  sufficient  to  execute  all  project  activities? 

9.  Are  the  technological  and  physical  infrastructures  adequate  to  support  all  project  activities? 

10.  Are  changing  circumstances  and  unpredictable  events  effectively  managed? 


Example  1:  Set  of  Drivers 
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Conducting  an 
Interview 


During  an  interview,  people  are  asked  a  series  of  questions  and  their 
responses  are  recorded.  Interview  questions  for  MDP  Activity  A1  are 
often  oriented  around  the  set  of  drivers.  Interview  participants  are 
asked  each  driver  question.  They  provide  their  response  to  each  ques¬ 
tion  as  well  as  their  rationale.  Example  2  is  an  example  set  of  data  for 
one  driver  question  gathered  from  an  interview  session. 


Question  Answer 


No 

Likely 

Equally 

Likely 

1 .  Are  project  goals  realistic 

no 

likely 

yes 

and  well-articulated? 

□ 

0 

□ 

□ 

Rationale 

There  were  goals  published  18  months  ago,  but  those  don't  seem  to  be  what 
managers  want  now.  I  have  not  seen  anything  recent.  I  have  heard  conflicting 
goals  from  the  hardware  lead  vs.  the  software  lead.  At  our  last  project  meet¬ 
ing  we  were  told  that  the  CIO  wants  this  delivery  schedule  decreased  by  6 
months  -  and  that  means  we  can't  meet  customer  requirements. 


Example  2:  Data  from  One  Person  for  One  Driver 
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3.3.2  GENERATE  DATA  FROM  DOCUMENTATION  (ACTIVITY  A2) 


INTRODUCTION  This  protocol  activity  (1)  collects  documents  that  are  relevant  to  the 

project  or  process,  such  as  policies,  procedures,  reports,  or  work  prod¬ 
ucts  and  (2)  generates  usable  data  from  that  documentation.  The  nature 
of  the  documentation  depends  upon  the  type  of  analysis,  the  specific 
project  or  process  being  assessed,  and  the  drivers  used  in  the  assess¬ 
ment.  For  example,  if  one  of  the  drivers  deals  with  efficient  workflow, 
then  the  design  of  the  process  may  be  reviewed. 

OBJECTIVES  This  activity  answers  the  following  questions: 

•  What  documentation  is  relevant  to  the  project  or  process? 

•  What  conditions  and  events  are  driving  the  project  or  process  to¬ 
ward  a  successful  outcome? 

•  What  conditions  and  events  are  driving  the  project  or  process  to¬ 
ward  an  unsuccessful,  or  failed,  outcome? 


3.3.2. 1  Dataflow 


Constraints 


Input 

12  Documentation 


A2 

Generate  data  from 
documentation 


Output 

->  N2  Data  from  documentation 


Resources 

Figure  9:  Inputs  and  Outputs  for  Activity  A2 


Input  and  Output 

Description 

12  Documentation 

Documentation  that  is  relevant  to  the  project  or  process.  Examples  include 
mission  statement,  policies,  procedures,  process  workflow,  work  products,  and 
quality  assurance  data. 

N2  Data  from  documenta¬ 
tion 

Usable  data  about  a  project  or  process  that  is  distilled  from  documentation, 
such  as  policies,  procedures,  work  products,  and  quality  assurance  data 

30  |  CMU/SEI-2008-TR-005 


3.3. 2. 2  Techniques 


Techniques 


Document 

Identification 


Document  Analysis 


The  following  two  classes  of  techniques  are  normally  employed  when 
conducting  this  protocol  activity:  (1)  techniques  used  to  identify  doc¬ 
uments  that  are  relevant  to  the  project  or  process  and  (2)  techniques 
used  to  analyze  those  documents  and  produce  data  that  are  pertinent  to 
subsequent  MDP  assessment  activities. 

The  goal  when  collecting  documents  is  to  gather  written  information, 
such  as  policies,  procedures,  reports,  and  work  products,  that  provide 
insight  into  how  a  project  or  process  is  managed  and  executed.  When 
you  gather  documentation,  you  can  follow  one  of  two  basic  strategies. 
You  can  develop  a  list  of  documents  that  you  want  to  review  and  then 
ask  people  to  provide  you  with  the  documents  on  the  list.  Alterna¬ 
tively,  you  can  ask  for  all  documentation  related  to  the  project  or  proc¬ 
ess  that  you  are  evaluating. 

You  also  have  options  related  to  when  you  ask  for  documentation.  You 
could  ask  for  documentation  during  Prepare  for  the  Assessment  (Phase 
1 )  when  you  are  getting  ready  to  conduct  the  evaluation.  This  approach 
allows  you  to  review  the  documents  before  you  meet  with  people  who 
work  on  the  project  or  process.  Alternatively,  you  can  ask  for  relevant 
documents  when  you  meet  with  people  during  Gather  Data  from  Peo¬ 
ple  (Activity  A1 ). 

Document  analysis  involves  reviewing  documentation  that  you  have 
gathered  during  the  assessment.  If  you  collect  all  of  the  documents  you 
originally  identified,  you  might  need  to  sort  through  the  documents  to 
determine  which  are  actually  relevant  to  the  assessment. 

When  you  review  a  given  document,  you  normally  frame  the  analysis 
around  an  explicit  set  of  guidelines  or  questions  (usually  related  to  the 
success  and  failure  drivers).  The  guidelines  or  questions  you  use  must 
be  appropriate  for  generating  sufficient  data  about  the  specific  subject 
or  problem  area  being  investigated.  Alternatively,  you  can  use  implicit 
guidelines  during  document  analysis.  Here,  you  use  your  expertise  and 
experience  to  look  for  data  that  would  be  useful  to  the  assessment. 
Overall,  this  technique  provides  a  first-pass  analysis  that  transforms 
raw,  unfiltered  information  into  data  that  are  usable  during  the  assess¬ 
ment. 
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3.3.2.3  Examples 


Key  QUESTIONS  FOR  Data  generated  from  analyzing  documents  are  usually  structured 
ANALYZING  around  the  driver  questions.  The  following  questions  are  examples  that 

DOCUMENTS  can  be  used  to  frame  the  document  analysis: 

•  What  information  from  the  document  supports  an  answer  of  yes  to 
each  driver  question? 

•  What  information  from  the  document  supports  an  answer  of  no  to 
each  driver  question? 

•  What  other  information  or  context  is  relevant  to  each  driver  ques¬ 
tion? 

Analyzing  a  Project 

pLAN  Example  3  depicts  the  results  for  one  driver  question  based  on  an  anal¬ 

ysis  of  a  project  plan. 


Document:  Project  Plan,  dated  August  25,  2007 
1 .  Are  project  goals  realistic  and  well-articulated? 

Information  supporting  an  answer  of  Yes 

Specific  project  goats  related  to  cost,  schedule,  quality,  and  customer  sat¬ 
isfaction  are  documented  in  the  project  plan  for  the  initial  release  and  the 
first  scheduled  update.  All  project  personnel  have  access  to  the  project 
plan. 

Information  supporting  an  answer  of  No 

The  schedule  has  been  reduced  since  the  plan  was  written.  The  goats  have 
not  been  adjusted  accordingly.  The  reduction  in  schedule  does  not  appear 
to  be  feasible  without  commensurate  adjustments  in  cost  and  quality. 


Other  relevant  data 


Example  3:  Driver  Analysis 
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3.3.3  EVALUATE  DRIVERS  (ACTIVITY  A3) 


INTRODUCTION  This  protocol  activity  evaluates  the  key  drivers  of  success  for  the  given 

project  or  process.  These  drivers  are  used  to  estimate  the  degree  of 
momentum  toward  project  or  process  objectives.  Each  driver  is  evalu¬ 
ated  against  a  set  of  criteria,  called  driver  evaluation  criteria,  to  deter¬ 
mine  its  effect  on  the  outcome.  Data  that  were  collected  from  people 
and  documentation  during  Activities  A1  and  A2  are  used  as  input 
when  evaluating  drivers. 

OBJECTIVES  This  activity  answers  the  following  questions: 

•  How  is  each  driver  affecting  the  outcome? 

•  Which  drivers  are  acting  as  success  drivers?  Why? 

•  Which  drivers  are  acting  as  failure  drivers?  Why? 


3.3.3. 1  Dataflow 


Constraints 


Inputs 

N1  Data  from  people 
N2  Data  from  documentation 


A3 

Evaluate 

drivers 


Output 

-►  01  Driver  values  &  rationale 


Resources 

Figure  10:  Inputs  and  Outputs  for  Activity  A3 


Inputs  and  Output 

Description 

N1  Data  from  people 

Usable  data  about  a  project  or  process  based  on  individual  and  group  perspec¬ 
tives,  information,  and  opinions  about  the  project  or  process  and  its  potential  for 

success 

N2  Data  from  documenta¬ 
tion 

Usable  data  about  a  project  or  process  that  is  distilled  from  documentation, 
such  as  policies,  procedures,  work  products,  and  quality  assurance  data 

01  Driver  values  and  ra¬ 
tionale 

The  current  status  of  each  driver,  which  includes 

•  the  driver  value 

•  rationale  that  explains  why  that  value  was  selected 
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3.3.3.2  Techniques 


TECHNIQUES  The  techniques  employed  when  conducting  this  protocol  activity  de¬ 

pend  upon  the  knowledge,  skills,  and  abilities  of  the  people  who  are 
performing  the  assessment.  Evaluating  drivers  generally  requires  tech¬ 
niques  for  analyzing  data  that  have  been  collected  during  earlier  activi¬ 
ties.  In  collaborative  settings  where  a  team  is  evaluating  drivers,  group 
decision-making  techniques  can  also  be  effective. 

DATA  ANALYSIS  Each  driver  represents  a  key  characteristic  of  a  project  or  process  that 

is  essential  for  a  successful  outcome.  When  you  evaluate  a  driver,  you 
must  rely  on  the  data  that  have  been  gathered  from  people  and  gener¬ 
ated  from  documentation.  Data  analysis  techniques  used  to  support  this 
protocol  activity  enable  you  to 

•  consider  the  range  of  possible  values  for  each  driver,  based  on  the 
relevant  value  criteria 

•  decide  which  value  is  most  appropriate  for  each  driver 

•  articulate  a  rationale  for  selecting  each  value 

When  evaluating  drivers  in  a  group  setting,  you  can  use  techniques  to 
facilitate  decision-making  activities.  For  example,  voting  techniques, 
such  as  multi-voting,  can  help  a  group  sort  through  differences  and 
reach  consensus. 


Group  Decision 
Making 


3. 3. 3. 3  Examples 

DRIVER  Values  Although  a  driver  is  characterized  by  a  yes/no  question,  some  degree 

of  ambiguity  or  uncertainty  regarding  the  answer  can  exist.  The  range 
of  answers  for  a  driver,  called  driver  values,  reflects  this  inherent  un¬ 
certainty.  Example  4  depicts  a  driver  question  and  its  range  of  values. 


Question 

No 

1 .  Are  project  goals  realistic 
and  well-articulated?  „ 


Answer 

Likely 

Equally 

Likely 

Yes 

no 

likely 

yes 

0 

□ 

□ 

□ 

Example  4:  Driver  and  Range  of  Values 
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DRIVER  Value  People  responding  to  driver  questions  need  to  interpret  the  underlying 

CRITERIA  meaning  of  each  possible  answer  (i.e.,  each  driver  value).  Driver  value 

criteria  provide  a  definition  for  each  value.  The  criteria  enable  people 
to  understand  what  is  implied  by  each  potential  answer  for  a  driver 
question.  Example  5  provides  an  example  set  of  criteria  that  define  the 
range  of  values  for  each  driver. 


Answer/Value 

Yes 

Likely  yes 

Equally  likely 

Likely  no 

No 


Definition 

The  answer  is  almost  certainly  “yes."  Almost  no  uncertainty  exists. 
There  is  little  or  no  possibility  that  the  answer  could  be  “no.” 

The  answer  is  most  likely  “yes.”  However,  a  degree  of  uncertainty 
exists.  There  is  some  possibility  that  the  answer  could  be  “no.” 

The  answer  is  just  as  likely  to  be  “yes”  or  “no.”  A  high  degree  of  un¬ 
certainty  exists. 

The  answer  is  most  likely  “no.”  However,  a  degree  of  uncertainty 
exists.  There  is  some  possibility  that  the  answer  could  be  “yes.” 

The  answer  is  almost  certainly  “no.”  Almost  no  uncertainty  exists. 
There  is  little  or  no  possibility  that  the  answer  could  be  “yes.” 


Example  5:  Criteria  for  Driver  Values 


TAILORING  Driver  The  criteria  for  evaluating  a  driver  can  be  tailored  for  the  given  as- 

VALUE  CRITERIA  sessment.  For  example,  the  criteria  in  Example  5  are  based  on  a  five- 

point  scale.  This  type  of  scale  allows  decision-makers  to  incorporate 
different  levels  of  uncertainty  in  their  answers.  More  or  less  than  five 
answers  can  be  incorporated  into  the  analysis  when  appropriate. 


EVALUATED  Drivers  Each  driver  is  evaluated  using  the  selected  scale  using  all  of  the  infor¬ 
mation  gathered  in  previous  activities,  including  the  driver  answers 
from  project  personnel.  In  Example  56  below,  the  following  final  an¬ 
swer  has  been  selected:  likely  no.  This  means  the  goals  being  pursued 
by  the  project  are  most  likely  unrealistic  or  not  well  understood  by 
staff.  The  rationale  for  the  answer  is  also  documented. 
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Question 

Answer 

1 .  Are  project  goals  realistic  and 
well-articulated? 

No 

Likely 

no 

Equally 

likely 

Likely 

yes 

Yes 

□ 

0 

□ 

□ 

□ 

Rationale 

-  The  schedule  for  integration  testing  is  likely  inadequate.  Historically,  inte¬ 
gration  testing  had  needed  considerably  more  buiid/test  cycles  for  similar 
applications. 

-  It  is  not  dear  whether  sufficient  contingency  plans  are  built  into  the  de¬ 
ployment  schedule;  potential  integration  issues  could  delay  deployment. 

-  Final  versions  of  applications,  components,  infrastructure,  and  training  used 
to  support  the  application  are  evolving;  they  might  not  be  compatible  and  like¬ 
ly  will  not  be  part  of  integration  testing. 

-  People  have  inconsistent  views  of  the  initial  deployment  goats  and  whether 
the  schedule  is  sufficient. 

-  The  initial  deployment  schedule  might  not  permit  enough  time  to  identify 
and  correct  problems  before  the  start  of  the  next  deployment. 


Example  6:  Evaluated  Driver 


DRIVER  Values  and  When  the  algorithm  used  to  assess  the  potential  for  success  employs 

SCORES11  simple  mathematics  (in  Activity  A4),  you  must  also  assign  a  number, 

called  a  driver  score ,  to  each  driver  value.  Example  7  illustrates  this 
concept.  Each  value  in  the  figure  represents  the  approximate  probabil¬ 
ity  that  the  driver  is  present.  For  this  example,  Likely  no  equates  to  a 
score  of  0.25,  which  represents  a  25%  likelihood  that  goals  are  realistic 
and  well  articulated. 


Question  Answer 


1 .  Are  project  goals  realistic 
and  well-articulated? 

No 

Likely 

no 

Equally 

likely 

Likely 

yes 

Yes 

□ 

0 

□ 

□ 

□ 

0 

0.25 

0.50 

0.75 

1.0 

Example  7:  Driver  Values  and  Scores 


11  It  might  seem  simpler  to  assign  numbers  directly  instead  of  answering  yes/no  answers.  However,  experience  shows  that 
many  people  have  trouble  assigning  numbers  directly.  These  people  find  it  easier  to  answer  the  question  “Are  project 
goals  realistic  and  well-articulated?”  with  “yes”  or  “no”  rather  than  translating  that  answer  into  a  number.  If  preferred,  you 
can  directly  assign  a  score  when  evaluating  drivers. 
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3.3.4  APPLY  ANALYSIS  ALGORITHM  (ACTIVITY  A4) 


INTRODUCTION  This  protocol  activity  estimates  a  project’s  or  process’  potential  for 

success  using  a  simple  algorithm.  In  an  MDP  assessment,  an  algorithm 
comprises  a  series  of  well-defined  instructions  for  estimating  the  po¬ 
tential  for  success  given  (1)  a  set  of  driver  values  or  scores  and  (2)  a  set 
of  predefined  success  criteria.  The  algorithm(s)  used  in  an  MDP  as¬ 
sessment  incorporate  simple  mathematics  or  rule -based  logic. 


OBJECTIVE  This  activity  answers  the  following  question: 

•  What  is  the  current  potential  for  success? 


3.3.4. 1  Dataflow 


Constraints 


Inputs 

N1  Data  from  people 
N2  Data  from  documentation 
01  Driver  values  &  rationale 


A4 

Apply  analysis 
algorithm 


Output 

02  Current  potential  for  success 


Resources 


Figure  11:  Inputs  and  Outputs  for  Activity  A4 


Inputs  and  Output 

Description 

N1  Data  from  people 

Usable  data  about  a  project  or  process  based  on  individual  and  group  perspec¬ 
tives,  information,  and  opinions  about  the  project  or  process  and  its  potential  for 

success 

N2  Data  from  documenta¬ 
tion 

Usable  data  about  a  project  or  process  that  is  distilled  from  documentation, 
such  as  policies,  procedures,  work  products,  and  quality  assurance  data 

01  Driver  values  and  ra¬ 
tionale 

The  current  status  of  each  driver,  which  includes 

•  the  driver  value 

•  rationale  that  explains  why  that  value  was  selected 

02  Current  potential  for 

success 

A  qualitative  measure  of  the  current  probability,  or  likelihood,  that  the  desired 
outcome  will  be  achieved  or  exceeded 
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3.3.4.2  Techniques 


Techniques 

The  following  two  types  of  algorithms  are  normally  employed  when 
conducting  this  protocol  activity:  (1)  algorithms  that  use  simple  ma¬ 
thematics  and  (2)  algorithms  that  use  rule-based  logic. 

Math-Based 

ALGORITHMS 

A  math-based  algorithm  employs  simple  mathematics  to  determine  the 
potential  for  success.  Before  using  this  type  of  algorithm,  you  must 
first  assign  a  driver  score,  or  number,  to  each  driver  value  in  the  set 
prior  to  using  the  algorithm.  The  algorithm  then  determines  the  poten¬ 
tial  for  success  using  the  driver  values  and  predefined  success  criteria. 
Math-based  algorithms  normally  rely  upon  one  or  more  of  the  follow¬ 
ing  basic  approaches: 

•  aggregate  driver  score 

•  weighted  aggregate  driver  score 

•  mean  driver  score 

•  median  driver  score 

Rule-Based 

ALGORITHMS 

This  technique  employs  rule -based  logic  to  determine  the  potential  for 
success.  A  set  of  rules  uniquely  defines  each  measure  in  the  predefined 
success  criteria  for  the  project  or  process.  This  technique  applies  the 
rules  embodied  in  the  success  criteria  to  a  set  of  driver  values  to  de¬ 
termine  the  potential  for  success. 
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3. 3. 4. 3  Examples 


SELECTING  A  Driver  As  part  of  tailoring  or  adapting  MDP  to  the  specific  needs  of  an  as- 

ANALYSIS  APPROACH  sessment,  you  need  to  select  an  analysis  approach.  Different  ap¬ 

proaches  for  analyzing  drivers  can  be  used  in  different  situations.  In 
general,  you  should  consider  the  following: 

•  the  goals  of  a  particular  assessment 

•  the  specific  drivers  being  used 

•  the  experience  and  expertise  of  people  conducting  the  assessment 

In  some  cases,  one  analysis  approach  will  be  sufficient  for  an  assess¬ 
ment.  In  other  instances,  you  might  decide  to  use  multiple  approaches 
to  analyze  a  set  of  drivers  (e.g.,  using  both  aggregate  driver  score  and 
rule-based  logic).  Employing  more  than  one  approach  provides  you 
with  multiple  views  of  the  data,  which,  in  some  instances,  can  enhance 
decision  making. 

Example  8  provides  a  set  of  driver  values  from  Activity  A3.  The  ex¬ 
amples  later  in  this  section  reference  the  driver  scores  in  Example  8. 


Driver  Score 

1 .  Are  project  goals  realistic  and  well-articulated?  0.25 

2.  Are  communication  and  information  sharing  about  project  activities  effective?  0 

3.  Are  customer  requirements  and  needs  well  understood?  0.5 

4.  Are  organizational  and  political  conditions  facilitating  completion  of  project  0 

activities? 

5.  Is  the  project  plan  sufficient?  0.25 

6.  Does  project  management  facilitate  execution  of  tasks  and  activities?  0.25 

7.  Is  task  execution  efficient  and  effective?  0.75 

8.  Is  staffing  sufficient  to  execute  all  project  activities?  0.25 

9.  Are  the  technological  and  physical  infrastructures  adequate  to  support  all  project  0.25 

activities? 

10.  Are  changing  circumstances  and  unpredictable  events  effectively  managed?  0.25 

Total  2.75 


10.  Are  changing  circumstances  and  unpredictable  events  effectively  managed?  0.25 

Total  2.75 


Example  8:  Evaluated  Driver  Set 
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SUCCESS  Criteria  Success  criteria  define  the  measures  used  to  characterize  the  potential 

for  success.  Each  measure  in  the  criteria  specifies  the  conditions  that 
must  be  satisfied  for  that  measure.  In  addition,  it  is  important  to  note 
that  all  measures  in  the  criteria  must  be  mutually  exclusive.  Success 
criteria  are  a  vital  part  of  MDP  because  they  enable  people  to  interpret 
a  set  of  drivers.  Example  9  depicts  success  criteria  that  can  be  used 
with  an  aggregate  driver  score.  When  you  compare  the  aggregate  driv¬ 
er  score  from  Example  8  to  the  success  criteria  in  Example  9,  you  can 
then  determine  that  the  potential  for  success  for  this  example  is  low. 


Measure 

Description 

Range 

Excellent 

Conditions  are  extremely  favorable  for  a  successful  outcome. 

8.5-10 

High 

Conditions  are  favorable  for  a  successful  outcome. 

6.5 -8.4 

Borderline 

Conditions  are  mixed,  making  the  outcome  uncertain. 

3.5 -6.4 

Low 

Conditions  are  not  favorable  for  a  successful  outcome. 

1.5 -3.4 

Minimal 

Conditions  are  extremely  unfavorable  for  a  successful  outcome. 


0  -1.4 

Example  9:  Success  Criteria 


TAILORING  Success  Success  criteria  must  be  tailored  appropriately  for  each  context  in 
CRITERIA  which  they  are  used.  When  tailoring  success  criteria,  you  must  con¬ 

sider  the  following  three  requirements: 

1 .  Success  criteria  must  be  suitable  for  the  analysis  approach  being 
used.  For  example,  the  criteria  in  Example  8  are  appropriate  for  an 
analysis  approach  using  aggregated  driver  scores. 

2.  The  measurement  scale  included  in  success  criteria  should  reflect 
the  needs  of  decision  makers.  A  two-point  scale  is  simple,  but  gen¬ 
erally  provides  insufficient  differentiation  for  decision-makers.  A 
twenty-point  scale  provides  considerable  differentiation  but  can  be 
difficult  to  use  with  any  degree  of  accuracy. 

3.  The  success  criteria  must  reflect  the  context  in  which  they  are  ap¬ 
plied.  For  example,  decision  makers  with  little  tolerance  for  risk 
may  want  a  narrower  range  for  the  measure  of  Excellent  (9.5  -  10). 
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Algorithms  in  MDP 


Examples  of  the  following  algorithms  are  presented  here: 


Aggregate  Driver 
Score 


Weighted 
Aggregate  Driver 
Score 


•  aggregate  driver  score 

•  weighted  driver  score 

•  mean  driver  score 

•  median  driver  score 

•  rule-based  logic 


The  aggregate  driver  score  is  the  sum  of  the  scores  for  all  drivers.  After 
you  determine  the  aggregate  driver  score,  you  then  compare  it  to  a  set 
of  predefined  success  criteria  (as  shown  in  Example  9)  to  determine  the 
potential  for  success.  An  analysis  approach  based  on  the  aggregate 
driver  score  is  a  straightforward  way  of  analyzing  a  set  of  drivers.  It 
allows  you  to  quickly  gauge  the  potential  for  success  based  on  the  rela¬ 
tive  influence  of  all  drivers.  Example  10  shows  the  aggregate  driver 
score  for  the  example  set  of  drivers  presented  in  Example  8. 


Aggregate  Driver  Score 

0.25  +  0  +  0.5  +  0  +  0.25  +  0.25  +  0.75  +  0.25  +  0.25  +  0.25  =  2.75 


Example  10:  Aggregate  Scoring 

To  determine  a  weighted  aggregate  driver  score,  you  assign  a  weight¬ 
ing  factor  to  each  driver.  Weighing  factors  are  based  on  the  relative 
influence  of  drivers  on  the  potential  for  success.  When  a  weighting 
factor  is  applied  to  a  driver,  its  score  is  multiplied  by  that  weighting 
factor.  A  total  weighted,  or  adjusted,  aggregate  driver  score  is  then 
calculated  by  adding  the  weighted  scores  for  all  drivers.  Example  1 1 
shows  an  example  of  applying  weighting  factors  where  drivers  1 ,  4, 
and  7  from  Example  8  are  judged  to  have  twice  the  influence  of  the 
other  drivers.  The  weighted  aggregate  driver  score  is  then  compared  to 
the  appropriate  success  criteria  (example  not  provided)  to  determine 
the  potential  for  success. 


Weighted  Driver  Score 

2(0.25)  +  0  +  0.5  +  2(0)  +  0.25  +  0.25  +  2(0.75)  +  0.25  +  0.25  +  0.25  =  3.75 


Example  11:  Weighted  Aggregate  Scoring 
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Effective 
Applications  of 
Weighting  Factors 


Mean  Driver  Score 


Median  Driver 
Score 


Effective  use  of  weighting  factors  requires  considerable  experience  and 
expertise  in  applying  MDP.  It  also  requires  you  to  have  a  considerable 
experience  with,  and  understanding  of,  the  project  or  program  being 
evaluated.  Without  appropriate  experience  and  expertise,  people  tend 
to  weight  drivers  improperly,  which  can  skew  results. 


The  mean  driver  score,  also  known  as  the  average  driver  score,  is  de¬ 
termined  by  adding  all  of  the  driver  values  and  dividing  by  the  number 
of  drivers.  Example  12  shows  the  mean  driver  score  for  the  set  of  driv¬ 
ers  presented  in  Example  8.  The  mean  value  for  the  set  of  drivers  is 
compared  to  the  appropriate  success  criteria  (example  not  provided)  to 
determine  the  potential  for  success. 


Mean  Driver  Score 

(0.25  +  0  +  0.5  +  0  +  0.25  +  0.25  +  0.75  +  0.25  +  0.25  +0.25)  /  10  =  0.275 


Example  12:  Mean  Scoring 

The  median  is  the  midpoint  in  a  series  of  numbers.  Half  of  the  numbers 
are  above  the  median  value,  while  the  other  half  are  below.  The  first 
step  when  determining  the  median  for  a  set  of  data  is  to  arrange  all  of 
the  numbers  in  order,  from  smallest  to  largest.  The  median  is  the  mid¬ 
dle  number  in  the  series.  Example  13  shows  the  data  from  Example  8 
arranged  from  smallest  to  largest.  In  this  example,  10  driver  scores  are 
included  in  the  series,  so  the  median  is  the  average  of  the  fifth  and 
sixth  scores.  The  median  driver  score  is  compared  to  the  appropriate 
success  criteria  (example  not  provided)  to  determine  the  potential  for 
success. 

Median  Driver  Score 

0  0  0.25  0.25  0.25  0.25  0.25  0.25  0.5  0.75 

(0.25  +  0.25)/2  =  0.25 


Example  13:  Median  Scoring 
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Algorithms  With 
Rule-Based  Logic 


Rule -based  algorithms  differ  from  the  other  approaches  profiled  in  this 
section  because  it  does  not  require  you  to  apply  mathematical  opera¬ 
tions  to  driver  scores.  Instead,  a  set  of  rules  uniquely  defines  each 
measure  in  the  success  criteria.  This  technique  applies  the  rules  em¬ 
bodied  in  the  success  criteria  to  a  set  of  driver  values  to  determine  the 
potential  for  success.  Each  measure  in  the  criteria  corresponds  to  one 
or  more  rules  that  specify  a  set  of  conditions  (related  to  the  drivers) 
unique  to  that  measure.  Consider  the  following  example  where  an  ex¬ 
cellent  potential  for  success  requires  meeting  the  following  two  rules: 

•  At  least  seven  driver  answers  must  be  yes. 

•  No  driver  can  have  an  answer  of  no,  likely  no,  or  equally  likely. 

Rules  for  other  measures  in  the  success  criteria  would  be  documented 
in  a  similar  manner.  This  analysis  approach  is  considerably  more  com¬ 
plex  than  the  others.  When  using  rule-based  logic,  you  must  ensure 
that  the  rules  for  all  measures  (1)  are  mutually  exclusive  and  (2)  ap¬ 
propriately  reflect  the  decision-makers  tolerance  for  risk. 
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3.3.5  ESTABLISH  SUCCESS  PROFILE  (ACTIVITY  A5) 


INTRODUCTION  This  protocol  activity  generates  a  success  profile  for  the  project  or 

process  by  (1)  setting  the  success  threshold,  (2)  comparing  the  current 
potential  for  success  to  the  success  threshold,  and  (3)  deciding  whether 
or  not  the  current  potential  for  success  is  acceptable.  The  success  thre¬ 
shold,  or  the  desired  potential  for  success,  represents  the  goal  for  the 
project  or  process  based  on  the  input  of  key  stakeholders. 

OBJECTIVES  This  activity  answers  the  following  questions: 

•  What  is  the  desired  potential  for  success  (i.e.,  success  threshold)  for 
the  project  or  process?  " 

•  What  is  the  gap  (success  differential)  between  the  current  potential 
for  success  and  the  success  threshold? 

•  What  conditions  and  potential  events  are  driving  the  gap  between 
the  current  potential  for  success  and  success  threshold?  How? 

•  To  what  extent  is  the  current  potential  for  success  acceptable? 


3.3.5. 1  Dataflow 


Constraints 


T 


Inputs 

N1  Data  from  people 

A5 

N2  Data  from  documentation  — 

► 

Establish 

01  Driver  values  &  rationale 

success  profile 

02  Current  potential  for  success 

X 


Output 

03  Success  profile 


Resources 


Figure  12:  Inputs  and  Outputs  for  Activity  A5 


12  While  it  may  seem  logical  to  assume  that  the  desired  potential  for  success  is  always  high,  that  is  not  always  true.  For 
example,  a  research  and  development  project  might  begin  with  a  low  potential  for  success.  A  research  and  development 
project  is  inherently  risky,  which  leads  to  a  low  potential  for  success.  However,  to  achieve  an  opportunity,  management 
might  decide  to  accept  the  low  potential  for  success.  People  must  use  common  sense  when  setting  the  desired  potential 
for  success.  Every  project  has  some  amount  of  uncertainty;  how  much  uncertainty  is  acceptable  to  stakeholders  helps 
determine  the  desired  potential  for  success. 
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Inputs  and  Output 

Description 

N1  Data  from  people 

Usable  data  about  a  project  or  process  based  on  individual  and  group  perspec¬ 
tives,  information,  and  opinions  about  the  project  or  process  and  its  potential  for 

success 

N2  Data  from  documenta¬ 
tion 

Usable  data  about  a  project  or  process  that  is  distilled  from  documentation, 
such  as  policies,  procedures,  work  products,  and  quality  assurance  data 

01  Driver  values  and  ra¬ 
tionale 

The  current  status  of  each  driver,  which  includes 

•  the  driver  value 

•  rationale  that  explains  why  that  value  was  selected 

02  Current  potential  for 

success 

A  qualitative  measure  of  the  current  probability,  or  likelihood,  that  the  desired 
outcome  will  be  achieved  or  exceeded 

03  Success  profile 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  for  success,  or  success  threshold 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold 

3.3.5.2  Techniques 

TECHNIQUES  The  following  types  of  techniques  are  used  when  establishing  a  success 

profile: 

•  establishing  the  success  threshold 

•  data  collection 

•  gap  analysis 

•  group  decision  making 
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Establishing  the 

Success  T  hreshold 

The  potential  for  success  characterizes  the  likelihood,  or  probability, 
that  the  desired  outcome  will  be  achieved  or  exceeded.  The  success 
threshold  is  the  desired,  or  target,  probability  for  the  project  or  process 
from  the  perspective  of  key  stakeholders  (e.g.,  a  15%  return  on  invest¬ 
ment,  90%  satisfied  customers,  80%  of  functional  requirements  deliv¬ 
ered).  It  reflects  stakeholders’  overall  tolerance  for  risk.  Techniques  for 
establishing  the  success  threshold  enable  you  to 

•  review  the  data  that  you  collected  from  each  key  stakeholder 

•  identify  which  stakeholders  are  the  key  decision  makers  for  the 
project  or  process 

•  determine  how  much  risk  or  uncertainty  key  decision  makers  will 

tolerate 

•  select  the  probability  that  most  appropriately  reflects  the  perspec¬ 
tive  of  key  stakeholders 

•  confirm  the  success  threshold  with  key  stakeholders  prior  to  per¬ 
forming  the  gap  analysis,  if  needed 

Data  Collection 

You  might  collect  all  of  the  data  you  need  to  establish  the  success  thre- 
shold  when  meeting  with  stakeholders  during  preparation.  Alterna¬ 
tively,  you  might  get  the  information  you  need  during  Activity  Al. 
Sometimes,  you  will  find  that  you  need  to  collect  additional  data  when 
you  are  ready  to  set  the  success  threshold  during  this  protocol  activity. 
You  can  use  several  techniques  to  collect  data  from  key  stakeholders. 
See  Activity  Al  for  a  summary  of  data  collection  techniques. 

Gap  Analysis 

Gap-analysis  techniques  are  used  to  compare  the  current  potential  for 
success  to  its  success  threshold.  These  techniques  are  useful  when  de¬ 
termining  whether  the  current  potential  for  success  is  acceptable  or  not. 
Gap-analysis  techniques  also  determine  which  conditions  are  contrib¬ 
uting  to  the  gap  and  how. 

Group  Decision 

Making 

When  analyzing  the  potential  for  success  in  a  group  setting,  you  can 
use  techniques  to  facilitate  decision-making  activities.  For  example, 
voting  techniques,  such  as  multi-voting,  can  help  a  group  sort  through 
differences  and  reach  consensus. 

13  The  success  threshold  must  be  set  by  the  time  this  activity  is  performed.  However,  you  can  set  the  success  threshold 
earlier  in  the  assessment,  for  example  during  the  Phase  1  preparation  activities,  based  on  information  gathered  from  sen¬ 
ior  managers  and  others. 
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3. 3. 5. 3  Examples 


SUCCESS  Criteria  Success  criteria  define  the  measures  used  to  characterize  qualitative 

measures  of  the  potential  for  success.  The  measures  are  used  in  Activ¬ 
ity  A4  to  determine  the  current  potential  for  success.  When  setting  the 
success  threshold,  you  are  determining  the  desired  potential  for  suc¬ 
cess.  This  example  depicts  success  criteria  using  the  aggregate  driver 
score  and  success  criteria  presented  in  Example  9.  The  criteria  from 
Example  9  are  shown  below  in  Example  14.  (See  section  3.3.4  for 
more  information  on  aggregate  driver  score.) 


Measure 

Description 

Range 

Excellent 

Conditions  are  extremely  favorable  for  a  successful  outcome. 

8.5-10 

High 

Conditions  are  favorable  for  a  successful  outcome. 

6.5  -  8.4 

Borderline 

Conditions  are  mixed,  making  the  outcome  uncertain. 

3.5 -6.4 

Low 

Conditions  are  not  favorable  for  a  successful  outcome. 

1.5  -  3.4 

Minimal 

Conditions  are  extremely  unfavorable  for  a  successful  outcome. 

0  -1.4 

Example  14:  Success  Criteria 

SETTING  the  Success  The  success  threshold  reflects  stakeholders’  overall  tolerance  for  risk. 

THRESHOLD  For  this  example,  management  and  other  stakeholders  discussed  their 

views  about  the  desired  potential  for  success.  In  many  cases,  managers 
focused  on  the  tangible  objectives  of  the  project,  for  example: 

•  Customer  satisfaction  is  a  primary  goal  of  the  project. 

Managers  were  adamant  about  meeting  the  schedule  with  a  high- 
quality  product  that  meets  at  least  95%  of  its  stated  functional  re¬ 
quirements. 

•  The  customer  for  this  project  is  critical  to  the  future  success  of  the 
company,  and  therefore  this  project  is  critical. 

The  project  being  assessed  is  a  straightforward  development  project 
that  incorporates  well  understood  technologies;  it  is  low  risk  from  a 
technical  point  of  view.  In  addition,  the  project  is  more  than  half-way 
through  its  development  life  cycle.  Based  on  the  nature  of  the  project, 
its  goals,  and  the  current  life  cycle  phase,  the  success  threshold  is  de¬ 
termined  to  be  high. 
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SUCCESS  Profile  FOR  The  example  below  depicts  a  success  profile.  The  success  threshold 
THE  PROJECT  desired  by  stakeholders  and  managers  was  high,  but  the  current  poten¬ 

tial  for  success  is  actually  low.  This  sizeable  gap  is  the  success  differ¬ 
ential  for  the  project. 


Excellent 


High  - 
Borderline  - 

Low  - 

Minimal  - 


Success 

differential 


-  Success  threshold 

(desired  potential  for  success) 


Current  potential  for  success 


Example  15:  Success  Profile 
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3.3.6  DETERMINE  NEXT  STEPS  (ACTIVITY  A6) 


INTRODUCTION  This  protocol  activity  identifies  actions  that  will  be  implemented  after 

the  assessment  to  maintain  or  improve  the  current  potential  for  success 
The  results  of  this  activity  serve  as  a  bridge  between  the  MDP  assess¬ 
ment  and  any  follow-on,  detailed  strategy  development  and  planning 
activities.  All  actions,  or  next  steps,  identified  during  this  protocol  ac¬ 
tivity  should  be  at  an  appropriate  level  of  detail  based  on  the  goals  of 
the  assessment,  depth  and  breadth  of  the  data  collected,  analysis  algo¬ 
rithm  used,  knowledge,  skills,  and  abilities  of  the  people  conducting 
the  assessment,  and  expectations  of  stakeholders.14 


OBJECTIVE  This  activity  answers  the  following  question: 

•  What  actions  will  help  maintain  or  improve  the  current  potential  for 
success? 

•  Who  is  responsible  for  each  action? 

•  By  when  must  each  action  be  completed? 


3.3.6. 1  Dataflow 


Constraints 


Inputs 

N1  Data  from  people 
N2  Data  from  documentation 
01  Driver  values  &  rationale 
03  Success  profile 


A6 

Determine 
next  steps 


Output 

04  Next  steps 


Resources 


Figure  13:  Inputs  and  Outputs  for  Activity  A6 


14  The  results  of  this  protocol  activity  can  range  from  a  simple  set  of  recommendations  or  list  of  action  items  to  a  detailed 
plan  that  includes  resource  estimates,  budget,  and  schedule. 
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Inputs  and  Output 

Description 

N1  Data  from  people 

Usable  data  about  a  project  or  process  based  on  individual  and  group  perspec¬ 
tives,  information,  and  opinions  about  the  project  or  process  and  its  potential  for 

success 

N2  Data  from  documenta¬ 
tion 

Usable  data  about  a  project  or  process  that  is  distilled  from  documentation, 
such  as  policies,  procedures,  work  products,  and  quality  assurance  data 

01  Driver  values  and  ra¬ 
tionale 

The  current  status  of  each  driver,  which  includes 

•  the  driver  value 

•  rationale  that  explains  why  that  value  was  selected 

03  Success  profile 

The  current  status  of  the  project  or  process,  including 

•  measure  of  the  current  potential  for  success 

•  measure  of  the  desired  potential  for  success,  or  success  threshold 

•  analysis  of  the  gap  between  the  current  potential  for  success  and  its  suc¬ 
cess  threshold 

04  Next  steps 

Actions  that  will  be  taken  after  the  assessment  is  complete 

3.3.6.2  Techniques 


Techniques 

Several  types  of  techniques  can  be  used  when  you  are  determining 
what  approach  to  take  after  the  assessment,  including 

•  action  planning 

•  brainstorming 

•  group  decision  making 

Action  Planning 

Action  planning  is  a  basic  technique  for  determining  how  to  proceed 
after  an  MDP  assessment  is  complete.  When  performing  this  tech¬ 
nique,  you  (1)  identify  a  candidate  list  of  actions,  or  next  steps,  (often 
using  brainstorming  techniques)  and  (2)  determine  which  of  the  candi¬ 
date  actions  will  be  implemented  after  the  assessment  is  complete.  The 
results  of  action  planning  lay  the  groundwork  for  subsequent  im¬ 
provement  activities. 

Brainstorming 

Brainstorming  is  a  basic  technique  for  generating  ideas.  It  can  be  used 
to  identify  a  candidate  list  of  actions  for  maintaining  or  improving  the 
current  potential  for  success.  Many  variants  of  brainstorming  exist  and 
can  be  used  when  performing  this  protocol  activity. 
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Group  Decision 
Making 


When  selecting  an  appropriate  set  of  next  steps,  you  can  use  techniques 
to  facilitate  decision-making  activities.  For  example,  voting  tech¬ 
niques,  such  as  multi-voting,  can  help  a  group  sort  through  differences 
and  reach  consensus. 


3. 3. 6. 3  Examples 

NEXT  Steps  For  the  example  project,  the  current  potential  for  success  is  low,  while 

the  desired  potential  for  success  is  high.  The  results  indicate  that  cur¬ 
rent  conditions  are  not  favorable  for  a  successful  outcome  and  that  the 
gap  between  current  and  desired  performance  is  large.  The  example 
below  illustrates  some  of  the  actions  selected  by  the  project  team.  De¬ 
tailed  planning  for  each  action  was  performed  outside  of  the  assess¬ 
ment  process. 


Action 

Responsibility 

Date 

Work  with  Center  Director  to  resolve  issues 
between  analysts  and  developers 

Project  Manager 

July  3 

Develop  and  implement  a  new  strategy  for  com - 
municating  information  with  project  team 

Project  Manager 

July  3 

Begin  a  major  replanning  effort  for  the  project 

Project  Manager  and 

July  12 

Technical  Leads 

- HI 

Example  16:  Next  Steps 
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3.4  COMPLETE  POST-ASSESSMENT  ACTIVITIES  (PHASE  3) 


INTRODUCTION  Phase  3  conveys  the  results  of  the  MDP  assessment  to  key  stakeholders 

and  identifies  actions  that  can  help  the  efficiency  and  effectiveness  of 
the  MDP  assessment.  The  objective  when  communicating  assessment 
results  to  stakeholders  is  to  present  findings  in  a  format  that  meets  their 
needs  and  requirements.  Different  formats  might  be  needed  to  commu¬ 
nicate  results  to  different  types  of  stakeholders. 

A  postmortem  is  used  to  identify  and  document  ways  in  which  the 
MDP  assessment  can  be  improved.15  Updates  and  improvements  to 
MDP  assessment  procedures,  artifacts,  tools,  and  training  are  made  as 
appropriate. 


OBJECTIVES  This  protocol  phase  answers  the  following  questions: 

•  Who  needs  to  know  the  results  of  the  assessment?16 

•  What  information  does  each  stakeholder  need? 

•  How  should  information  be  communicated  to  each  stakeholder? 

•  What  lessons  were  learned  when  preparing  for  the  assessment? 

•  What  lessons  were  learned  when  conducting  the  assessment? 

•  How  do  the  assessment  procedures,  artifacts,  tools,  and  training 
need  to  be  updated  or  improved? 


DATAFLOW  The  following  diagram  highlights  the  dataflow  for  this  protocol  phase. 

Constraint 

Cl  Assessment  constraints 


Inputs 

PAI1  MDP  assessment  results  &  plans 
PAI2  MDP  assessment  procedures, 
artifacts,  tools,  &  training 


Phase  3 
Complete  Post- 
Assessment 
Activities 


Outputs 

PAOI  Communicated  assessment  results 
PA02  Lessons  learned 
PA03  Updates  to  MDP  assessment 
procedures,  artifacts,  tools,  &  training 


Resources 

R5  Experienced  personnel 
R6  Post-assessment  procedures 
R7  Post-assessment  artifacts  &  tools 


Figure  14:  Dataflow  for  MDP  Phase  3 


15  Postmortems  are  usually  conducted  after  a  given  assessment.  However,  they  can  also  be  held  on  a  more  periodic  basis  if 
multiple  assessments  are  planned. 

16  Requirements  for  communicating  assessment  results  are  part  of  the  assessment  plan  that  is  developed  in  Phase  1 . 

These  requirements  are  revisited  in  Phase  3  and  are  revised  when  appropriate  (e.g.,  if  new  stakeholders  are  identified 
during  the  assessment). 
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Inputs 


The  following  inputs  are  required  by  the  activities  performed  during 
this  protocol  phase. 


Type 

Description 

PAI1  MDP  assessment 
results  and  plans 

All  outputs  produced  by  the  MDP  assessment,  including  findings  and  assess¬ 
ment  data,  as  well  as  plans,  budget,  and  schedule  for  conducting  the  assess¬ 
ment 

PAI2  MDP  assessment 
procedures,  artifacts,  tools, 
and  training 

Supporting  materials  used  to  conduct  an  MDP  assessment,  including  proce¬ 
dures,  worksheets,  databases,  and  training  artifacts 

CONSTRAINT  The  following  constraint  affects  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

Cl  Assessment  constraints 

Any  circumstances,  including  logistics,  personnel,  schedule,  and  cost  issues, 
that  could  affect  assessment  activities 

RESOURCES  The  following  resources  support  execution  of  the  activities  performed 

during  this  protocol  phase. 


Type 

Description 

R5  Experienced  personnel 

People  who  are  experienced  in  all  phases  of  an  MDP  assessment 

R6  Post-assessment  pro¬ 
cedures 

Documentation  that  describes  how  to  conduct  post-assessment  activities 

R7  Post-assessment  arti¬ 
facts  and  tools 

Templates,  worksheets,  standard  presentations,  automated  tools,  and  data¬ 
bases  needed  to  conduct  post-assessment  activities 
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Outputs 


The  following  outputs  are  produced  by  the  activities  performed  during 
this  protocol  phase. 


Type 

Description 

PAOI  Communicated  as¬ 
sessment  results 

Assessment  results  that  have  been  conveyed  to  key  stakeholders,  including 

•  driver  values 

•  success  profile  for  the  project  or  process 

•  actions  that  need  to  be  implemented  to  maintain  or  improve  the  current 
potential  for  success 

•  supporting  data  as  appropriate 

PA02  Lessons  learned 

Knowledge  gained  by  conducting  an  MDP  assessment  that  can  be  used  to 
modify  and  improve  future  MDP  assessments 

PA03  Updates  to  MDP 
assessment  procedures, 
artifacts,  tools,  and  training 

Any  changes,  based  on  lessons  learned,  to  MDP  assessment  procedures, 
artifacts,  tools,  and  training  intended  to  improve  the  efficiency  and  effectiveness 
of  future  MDP  assessments 

Key  ACTIVITIES  The  following  table  highlights  the  activities  performed  during  this  pro¬ 

tocol  phase.17 


Activity 

Description 

Communicate  results 

Convey  the  results  of  the  MDP  assessment  to  key  stakeholders 

Conduct  postmortem  of  the 
MDP  assessment 

Conduct  one  or  more  meetings  to  identify  the  strengths  and  weaknesses  of  the 
MDP  assessment  and  document  modifications  and  improvement  to  the  MDP 
assessment  process 

Implement  improvements  to 
the  MDP  assessment 

process 

Make  changes,  based  on  lessons  learned,  to  the  MDP  assessment  process, 
including  updating  procedures,  artifacts,  tools,  and  training  as  appropriate 

17  Detailed  descriptions  of  Phase  3  activities  are  not  provided  in  this  document. 
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4  Summary  and  Further  Work 


MISSION  Success  In  2006,  the  Carnegie  Mellon®  SEI  chartered  the  Mission  Success  in 

RESEARCH  and  Complex  Environments  (MSCE)  project  to  advance  the  risk- 

DEVELOPMENT  management  state -of-the -practice.  A  key  part  of  this  project  is  the 

development  of  MOSAIC — a  suite  of  risk-based  methods  for  assess¬ 
ing  and  managing  complex  projects  and  processes. 

An  extensive  and  time-consuming  analysis  is  normally  required  when 
conducting  most  commonly  used  risk  assessments.  An  underlying 
goal  of  the  SEI  research  and  development  activities  related  to  MDP  is 
to  demonstrate  that  an  assessment  does  not  require  a  significant  in¬ 
vestment  of  time  to  be  effective.  In  this  way,  people  can  more  readily 
adopt  risk-based  approaches  that  help  them  weigh  the  alternatives 
confronting  them  and  ultimately  make  better  decisions. 

MDP  is  the  first  MOSAIC  assessment  protocol  to  be  published. 

Please  refer  to  the  MSCE  web  site  for  current  information: 

http://www.sei.cmu.edu/msce/ 


MDP  IS  A  RISK-BASED  MDP  is  a  risk-based  assessment  for  evaluating  current  conditions, 
APPROACH  determining  a  project’s  or  process’  current  potential  for  success,  and 

actions  that  will  help  maintain  or  improve  the  current  potential  for 
success  over  time.  It  can  be  applied  to  projects  and  processes  across 
the  life  cycle  and  throughout  the  supply  chain  and  is  designed  to  help 
people  analyze  tradeoffs  and  make  better  decisions  in  situations  that 
have  a  high  degree  of  uncertainty.  When  used  properly,  an  MDP  as¬ 
sessment  provides  a  time-  and  resource-efficient  way  to  identify  ma¬ 
jor  issues  that  can  affect  the  potential  for  success. 

A  small  set  of  drivers  is  used  to  gauge  the  current  conditions  affecting 
a  project  or  process.  Then,  a  simple  algorithm  is  applied  to  the  set  of 
drivers  to  determine  the  degree  of  momentum  toward  the  desired  out¬ 
come,  which  allows  for  actionable  assessment  results  without  having 
extensive  expertise  in  conducting  assessments.  However,  an  MDP 
assessment  only  provides  a  “ballpark”  measure  of  the  potential  for 
success.  It  can  be  viewed  as  a  first-pass  screening  of  a  project  or 
process  to  diagnose  any  unusual  circumstances  that  might  affect  its 
potential  for  success.  More  detailed,  follow-on  evaluations  might  be 
required  when  the  potential  for  success  is  judged  to  be  unacceptable. 


Carnegie  Mellon  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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MDP  Pilots 


MOSAIC 

Management 

Approach 


MDP  was  designed  for  use  in  many  different  domains  and  types  of 
problems.  To  date,  MDP  has  been  piloted  in  the  following  domains: 

•  cyber-security  incident  response 

•  project  portfolio  management 

•  software  development 

Early  piloting  of  MDP  has  demonstrated  its  flexibility  by  showing  how 
it  can  be  applied  in  a  variety  of  domains  and  environments,  across  the 
life  cycle,  and  throughout  the  supply  chain. 


MOSAIC  requires  establishing  and  maintaining  a  reasonable  degree  of 
confidence  that  objectives  will  be  achieved  successfully.  Figure  15 
depicts  the  key  activities  performed  when  using  MOSAIC  to  manage  a 
project  or  process.  Notice  that  an  assessment  is  a  key  activity  of  this 
approach.  However,  assessing  the  current  potential  for  success  (using 
MDP  for  example)  is  just  one  part  of  the  broader  management  ap¬ 
proach.  Additional  follow-on  activities  are  needed  to  help  ensure  that 
the  desired  outcome  will  be  achieved. 
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MOSAIC 

Management 

Activities 


Future  MDP 
Development 


MSCE  Research 
Directions  and  Goal 


As  illustrated  in  Figure  15,  MOSAIC  requires  completing  the  follow¬ 
ing  key  activities: 

•  Assess — determine  the  current  potential  for  success  in  relation  to  its 
success  threshold 

•  Plan — develop  a  detailed  action  plan  for  maintaining  or  improving 
the  potential  for  success  over  time 

•  Implement — execute  plans  as  defined 

•  Track — monitor  the  status  of  plan  milestones  and  measures  of  plan 
effectiveness 

•  Control — make  adjustments  to  plans  when  appropriate 

MDP  enables  you  to  assess  the  current  potential  for  success.  In  addi¬ 
tion,  it  also  kicks  off  the  planning  activity  by  identify  how  to  proceed 
after  the  assessment  is  complete.  Additional  follow-on  planning  is 
normally  required  to  develop  a  formal  improvement  plan. 


MDP  is  an  important  piece  of  research  because  it  provides  a  founda¬ 
tion  for  future  research  and  development  activities  related  to 
MOSAIC.  We  intend  to  continue  piloting  MDP  in  different  venues. 
We  also  intend  to  publish  guidebooks  focusing  on  how  to  conduct  an 
MDP  assessment,  and,  when  appropriate,  we  will  also  develop  do- 
main-specific  methods  consistent  with  MDP. 


We  intend  to  continue  developing  the  MOSAIC  suite  of  assessment 
and  management  methods.  The  early  focus  of  MOSAIC  research  has 
been  on  assessing  the  potential  for  success.  Future  research  will  focus 
on  developing  approaches  for  managing  the  potential  for  success  over 
time. 

Overall,  the  main  goal  of  our  research  is  to  transform  risk  management 
from  a  hazard-driven  discipline  to  a  success-  and  opportunity-driven 
discipline.  Our  work  with  MDP  is  the  first  step  toward  achieving  that 
goal. 
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Appendix  A:  Risk  Management  Concepts 


Multiple  Contexts 

of  Risk  Management 

The  term  risk  is  used  universally,  but  different  audiences  often  attach 
different  meanings  to  it  [Kloman  90].  In  fact,  the  details  about  risk  and 
how  it  supports  decision  making  depend  upon  the  context  in  which  it 
is  applied  [Charette  90].  For  example,  safety  professionals  view  risk 
management  in  terms  of  reducing  the  number  of  accidents  and  injuries. 
A  hospital  administrator  views  risk  as  part  of  the  organization’s  qual¬ 
ity  assurance  program,  while  the  insurance  industry  relies  on  risk  man¬ 
agement  techniques  when  setting  insurance  rates.  Each  industry  thus 
uses  a  definition  that  is  uniquely  tailored  to  its  context.  No  universally 
accepted  definition  of  risk  exists. 

Three 

Characteristics  of 

Risk 

Whereas  specific  definitions  of  risk  might  vary,  a  few  characteristics 
are  common  to  all  definitions.  For  risk  to  exist  in  any  circumstance, 
the  following  three  conditions  must  be  satisfied  [Charette  1990]: 

1 .  The  potential  for  loss  must  exist. 

2.  Uncertainty  with  respect  to  the  eventual  outcome  must  be  pre- 

,  is 

sent. 

3.  Some  choice  or  decision  is  required  to  deal  with  the  uncertainty 
and  potential  for  loss. 

Three  Conditions  of 

Risk 

The  three  characteristics  can  be  used  to  forge  a  very  basic  definition  of 
the  word  risk.  Most  definitions  focus  on  the  first  two  conditions — loss 
and  uncertainty — because  they  are  the  two  measurable  aspects  of  risk. 
Thus,  the  essence  of  risk,  no  matter  what  the  domain,  can  be  succinctly 
captured  by  the  following  definition:  Risk  is  the  possibility  of  suffering 
loss  [Dorofee  1996]. 

18  Some  researchers  separate  the  concepts  of  certainty  (the  absence  of  doubt),  risk  (where  the  probabilities  of  alternative 
outcomes  are  known),  and  uncertainty  (where  the  probabilities  of  possible  outcomes  are  unknown).  However,  because 
uncertainty  is  a  fundamental  attribute  of  risk,  we  do  not  differentiate  between  decision  making  under  risk  and  decision 
making  under  uncertainty  in  this  technical  report. 
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Speculative  and 
Hazard  Risk 


Speculative  Risk 
Example:  Gambling 


Sometimes  a  situation  presents  an  opportunity  for  gain  as  well  as  the 
potential  for  loss.  In  other  instances,  only  the  potential  for  loss  exists. 
Because  of  this  difference,  risk  can  thus  be  further  subdivided  into  two 
types:  speculative  risk  and  hazard  risk  [Young  2001].  Figure  16  graph¬ 
ically  illustrates  the  difference  between  speculative  and  hazard  risk. 

With  speculative  risk  you  might  realize  a  gain,  which  can  improve 
your  current  situation  relative  to  the  status  quo.  At  the  same  time,  you 
might  experience  a  loss,  which  can  make  your  situation  worse  than  it 
is  at  present.  In  contrast,  hazard  risk  provides  no  opportunity  to  im¬ 
prove  upon  the  current  situation;  it  only  brings  the  potential  for  loss. 


Figure  16:  Speculative  and  Hazard  Risks 

Gambling  is  an  example  of  taking  a  speculative  risk.  When  you  place  a 
bet,  you  must  balance  the  potential  for  gain  against  the  potential  for 
loss.  You  weigh  the  possibility  of  gaining  additional  money  against  the 
prospect  of  losing  the  funds  you  wagered.  When  you  gamble,  your 
objective  is  to  increase  your  wealth  in  relation  to  the  status  quo,  and 
you  are  willing  to  put  your  finances  at  risk  for  the  opportunity  to  make 
money. 
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Speculative  Risk 
Example:  Business 
Risk 


Hazard  Risk 
Example:  Security 


Speculative  Risk 
Example:  Security 


Importance  of 
Context 


Business  risk  is  another  example  of  speculative  risk.  When  managers 
invest  organizational  assets,  they  must  balance  the  risk  of  investing 
organizational  capital  against  the  potential  return  on  that  investment. 
From  an  economic  perspective,  as  an  organization’s  risk  increases,  its 
potential  return  on  investment  had  better  increase  correspondingly. 
Management  should  never  take  on  additional  risk  unless  the  potential 
for  increased  profits  also  exists.  The  balance  of  risk  and  opportunity 
drives  all  business  decisions,  which  makes  business  risk  speculative. 


Consider  how  security  can  be  viewed  as  a  hazard  risk.  Imagine  that 
you  are  concerned  about  protecting  valuables  that  are  stored  in  your 
home.  Your  main  objective  in  this  example  is  to  ensure  that  none  of 
the  valuables  in  your  residence  is  removed  without  your  knowledge 
and  permission.  After  evaluating  how  well  your  valuables  are  pro¬ 
tected,  you  might  decide  to  install  a  security  system  in  your  residence 
to  make  it  more  difficult  for  a  thief  to  break  in  and  steal  your  valu¬ 
ables.  Notice  that  the  objective  in  this  example,  by  definition,  restricts 
the  focus  of  risk  on  the  potential  for  loss.  In  the  most  favorable  of  cir¬ 
cumstances,  you  only  keep  what  you  already  possess.  There  is  no  po¬ 
tential  for  gain. 


Now  consider  the  same  example  when  viewed  from  another  perspec¬ 
tive.  In  this  instance,  you  would  like  to  gain  peace  of  mind  by  prevent¬ 
ing  unsavory  characters  from  gaining  entrance  to  your  house.  Your 
objective  to  feel  more  secure  defines  the  context  in  which  you  view 
risk.  After  analyzing  the  situation,  you  might  decide  to  install  a  secu¬ 
rity  system  in  your  residence  to  make  it  more  difficult  for  someone  to 
break  in.  You  might  reason  that  the  added  protection  will  make  you 
feel  more  secure  and  help  you  gain  the  peace  of  mind  you  seek.  In  this 
example,  you  are  willing  to  invest  money  in  a  security  system  to  pro¬ 
vide  yourself  an  opportunity  feel  more  secure.  The  security  risk  in  this 
example  is  speculative  because  it  balances  your  tolerance  for  risk  (i.e., 
the  amount  of  money  you  are  willing  to  invest  in  a  security  system) 
with  your  desire  to  realize  an  opportunity  (i.e.,  gaining  peace  of  mind). 


The  two  security  examples  illustrate  how  the  same  situation  can  be 
viewed  as  a  hazard  risk  in  one  context  and  a  speculative  risk  in  an¬ 
other.  A  risk  therefore  is  classified  as  speculative  or  hazard  based  on 
the  context  in  which  it  is  viewed.  The  notion  of  explicitly  establishing 
the  context  in  which  you  analyze  and  manage  risk  is  vitally  important 
to  ensure  that  you  make  appropriate  choices  about  how  you  manage 
your  risk. 
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Five  Common 
Elements 


All  forms  of  risk,  whether  they  are  classified  as  speculative  or  hazard 
risk,  comprise  common  elements.  This  notion  is  illustrated  in  Figure 
17,  which  highlights  the  following  five  common  elements  of  risk:  (1) 
context,  (2)  execution,  (3)  conditions,  (4)  potential  events,  and  (5) 
range  of  potential  outcomes. 


Context 


! 


Conditions 


Potential 

events 


Execution 


\ 

Range  of  1 
potential 
outcomes 

/ 


Figure  17:  Common  Elements  of  Risk 

CONTEXT  Context  provides  the  background,  situation,  or  environment  in  which  a 

project  or  process  is  executed.  It  generally  includes  the  key  objectives 
being  pursued  as  well  as  stakeholders’  expectations  for  those  objec¬ 
tives.  19  It  defines  the  picture  of  success  for  a  given  set  of  objectives 
and  provides  the  lens  through  which  all  potential  outcomes  are  viewed 
and  interpreted.  Defining  the  context  is  thus  an  essential  first  step 
when  managing  any  type  of  risk. 


EXAMPLE:  Project  Assume  that  you  are  a  project  manager  who  is  overseeing  the  devel- 
MANAGEMENT  opment  of  a  software-intensive  system.  Suppose  that  these  are  the 

CONTEXT  most  important  objectives  to  you:  product,  cost,  and  schedule.  These 

objectives  indicate  that  you  are  focused  on  developing  a  fully  func¬ 
tional  system  on  time  and  within  budget.  Now,  suppose  that  stake¬ 
holders  (such  as  senior  managers  in  your  organization)  are  very  con¬ 
cerned  about  cost  overruns  and  have  made  it  clear  that  the  project 
cannot  exceed  its  budget.  As  a  result,  the  cost  objective  becomes  your 
primary  objective  among  the  three,  and  your  tolerance  for  cost  risk  is 
low.  Your  decisions  will  be  driven  by  your  low  tolerance  for  cost  over¬ 
runs.  When  you  are  forced  to  make  tradeoffs,  unacceptable  outcomes 
related  to  cost  will  have  a  greater  influence  than  those  related  to  prod¬ 
uct  and  schedule  objectives. 


19  Stakeholders  include  all  interested  parties,  customers,  and  suppliers,  both  internal  and  external  to  an  organization. 
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Foundation  of  Risk 

Management 

The  context  in  the  above  example  has  been  defined  by  three  project 
objectives  and  the  expectations  related  to  those  objectives.  Without 
setting  an  appropriate  context,  you  cannot  definitively  determine  how 
to  gauge  the  potential  for  success  or  how  to  assess  any  given  outcome. 
Context  thus  forms  the  underlying  foundation  when  managing  risk. 

Execution 

Execution  describes  what  must  be  done  to  achieve  a  set  of  objectives. 
With  respect  to  a  project  or  process,  execution  refers  to  the  activities 
that  are  performed  when  working  toward  the  objectives. 

Conditions 

Conditions  define  the  circumstances  that  directly  or  indirectly  influ¬ 
ence  execution  and  drive  an  outcome  toward  success  or  failure.  As  a 
project  or  process  is  executed,  these  conditions  affect  the  eventual  out¬ 
come.  In  some  instances,  conditions  directly  influence  the  outcome; 
while  in  others,  they  indirectly  affect  the  outcome  by  creating  expo¬ 
sure  to  negative  or  positive  events. 

Example:  Conditions 

that  Directly 

Influence  an 

Outcome 

Consider  an  example  where  a  team  is  developing  a  software-intensive 
system.  Suppose  that  the  following  condition  is  present:  team  mem¬ 
bers  have  not  previously  worked  with  the  design  language  being  used 
on  the  project.  This  could  cause  them  to  make  mistakes  or  take  more 
time  when  working  on  tasks,  driving  product,  cost,  and  schedule  ob¬ 
jectives  toward  one  or  more  undesired  outcomes.  Here,  the  condition 
has  a  direct  influence  on  the  eventual  outcome. 

Exposing  Conditions 

Conditions  that  expose  a  project  or  process  to  the  effects  of  events  that 
might  (or  might  not)  occur  are  called  exposing  conditions.  During 
normal  day-to-day  operations,  these  conditions  lie  dormant  and  do  not 
produce  any  visible  effect  on  results.  However,  certain  events  in  com¬ 
bination  with  exposing  conditions  can  influence  the  expected  out- 
20 

come. 

Potential  Events 

A  potential  event  is  an  unpredictable  occurrence  that  combines  with 
one  or  more  exposing  conditions  to  affect  performance  and  thus  drive 
the  outcome  toward  success  or  failure. 

20  Events  can  have  a  positive  or  negative  effect  on  the  outcome  depending  on  the  specific  nature  of  the  event.  For  example, 
an  increase  in  funding  would  likely  be  perceived  as  a  positive  event  that  might  put  a  project  in  better  position  for  success. 
On  the  other  hand,  a  decrease  in  funding  would  likely  be  perceived  as  a  negative  event  that  might  adversely  affect  a  pro¬ 
ject’s  outcome. 
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Example:  Potential 

Events  and  Exposing 

Conditions 

A  computer  virus  is  a  program  that  is  designed  to  exploit  certain  ex¬ 
posing  conditions  (called  vulnerabilities)  and  infect  computers  causing 
them  to  act  erratically.  People  with  malicious  intent  design  these  pro¬ 
grams  with  the  ultimate  goal  of  wreaking  havoc  throughout  the  busi¬ 
ness  community,  such  as  degrading  the  performance  of  computers  and 
networks  or  rendering  them  unavailable  for  use.  If  a  work  process  is 
highly  dependent  on  the  availability  of  computers  and  networks  that 
become  infected,  production  can  be  temporarily  halted,  which  can  lead 
to  an  undesirable  outcome."  Notice  that  the  condition,  the  system’s 
vulnerability,  poses  no  threat  to  production  during  normal  operations. 

It  takes  an  unpredictable  event,  the  proliferation  of  a  computer  virus, 
for  damage  to  occur.  This  particular  condition  only  affects  the  process’ 
outcome  when  a  relevant  event  occurs. 

Range  of  Potential 

Outcomes 

The  range  of  potential  outcomes  defines  the  set  of  possible  results  that 
can  be  achieved  when  executing  a  project  or  process.  Some  outcomes 
will  be  considered  to  be  acceptable,  while  others  will  be  viewed  as 
unacceptable. 

Traditional  Risk 

Management 

Approaches 

Most  risk-management  approaches,  when  applied  to  projects  and  proc¬ 
esses,  have  traditionally  assumed  a  hazard  view  of  risk.  From  the  haz¬ 
ard  perspective,  a  risk  is  viewed  as  a  potential  obstacle  that  can  inter¬ 
fere  with  positive  momentum  or  progress,  and  a  threat  is  defined  as  a 
condition  or  event  that  could  lead  to  a  risk  [Alberts  2005].  When 
viewed  from  this  perspective,  traditional  risk  management  focuses  on 
reducing  or  eliminating  obstacles  that  might  interfere  with  momentum 
or  progress.  In  addition,  traditional  risk  management  approaches  have 
not  considered  multiple  organizations;  they  focus  within  an  organiza¬ 
tion  and  locally  optimize  risk  for  that  organization. 

Focus  on  Single 

Conditions  or 

Events 

Traditional  risk-management  approaches,  when  applied  to  projects  and 
processes,  focus  on  individual  conditions  or  potential  events.  A  risk 
analysis  is  then  used  to  estimate  the  potential  consequence  triggered 
by  each  condition  or  event. 

21  Undesirable  from  the  business'  perspective,  that  is.  From  the  virus  developer’s  perspective,  this  would  be  considered  a 
successful  outcome. 
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Risk  Statement 


A  risk  is  normally  represented  using  a  linear  cause-and-effect  pair  that 
conveys  two  key  pieces  of  information:  (1)  the  threat  (i.e.,  condition  or 
potential  event)  that  is  causing  concern  and  (2)  the  potential  conse¬ 
quences  of  that  threat  [Gluch  1994].  Each  cause-and-effect  pair,  or  risk 
statement,  can  be  viewed  as  a  scenario  that  documents  the  potential 
loss  triggered  by  a  given  condition  or  event.  Figure  18  illustrates  the 
notion  of  multiple  risks  that  can  affect  a  project  or  process.  The  list  of 
risks  becomes  the  focal  point  of  risk  management  activities  in  tradi¬ 
tional  approaches. 


Basic  Risk 

Management 

Activities 


Figure  18:  Cause  and  Effect  Risk  Statement 

Traditional  risk-management  approaches  generally  require  people  to 
conduct  the  following  types  of  activities: 

•  identify  risks  that  can  lead  to  loss 

•  prioritize  the  list  of  risk  statements  based  on  objectives,  require¬ 
ments,  and  constraints 

•  develop  mitigation  plans  for  the  highest  priority  risks 

•  implement  the  mitigation  plans  as  defined 

•  track  the  status  of  mitigation  plan  milestones  and  measures  of  ef¬ 
fectiveness 

•  make  adjustments  to  mitigation  plans  when  appropriate 
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Appendix  B:  Protocol  Structure  and  Nomenclature 


Information 
for  all  Phases 


Information 
for  Phase  2 
Activities 


Table  1  describes  the  information  provided  in  this  report  for  all  phases 
of  the  Mission  Diagnostic  Protocol. 


Information  Type 

Description 

Introduction 

A  brief  introduction  describing  the  key  aspects  of  the  phase 

Objectives 

Key  objectives  for  the  phase  worded  as  questions 

Dataflow 

A  high-level  dataflow  diagram  for  the  protocol  phase 

Note:  For  Phase  2  of  an  assessment  protocol,  a  detailed 
dataflow  of  all  activities  is  also  provided. 

Inputs 

Data  that  are  required  by  a  protocol  phase 

Constraints 

The  limitations  imposed  on  a  protocol  phase  or  activity 

Resources 

Procedures,  plans,  artifacts,  tools,  people,  and  other  re¬ 
sources  that  support  execution  of  a  protocol  phase 

Outputs 

Data  that  are  produced  by  a  protocol  phase 

Key  activities 

A  brief  description  of  the  activities  performed  during  the 
phase 

Table  1:  Information  Types  for  all  Assessment  Phases 


Table  2  describes  the  information  provided  for  each  Phase  2  activity. 
The  same  constraints  and  resources  apply  to  all  Phase  2  activities. 
Therefore,  descriptions  of  constraints  and  resources  are  presented  in 
Phase  2  and  are  not  repeated  for  each  individual  Phase  2  activity. 


Information  Type 

Description 

Introduction 

A  brief  introduction  describing  the  key  aspects  of  the  proto¬ 
col  activity 

Objectives 

Key  objectives  for  the  protocol  activity  worded  as  questions 

Dataflow 

A  high-level  dataflow  diagram  for  the  protocol  activity 

Inputs 

Data  that  are  required  by  a  protocol  activity 

Outputs 

Data  that  are  produced  by  a  protocol  activity 

Techniques 

A  brief  description  of  the  types  of  techniques  that  can  be 
used  to  conduct  the  protocol  activity 

Table  2:  Information  Types  for  each  Phase  2  Activity 
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Dataflow 

Structure 


Dataflow 

Identifiers 


Figure  19  depicts  the  data  types  included  in  a  protocol  dataflow.  The 
same  data  types  are  used  when  documenting  the  dataflow  for  a  proto¬ 
col  phase  or  an  activity. 


Constraints 


Inputs 


Protocol  Phase 
or  Activity 


->  Outputs 


Resources 


Figure  19:  Protocol  Data  Types 


Each  input,  output,  constraint,  and  resource  listed  in  a  dataflow  is  rep¬ 
resented  by  an  identifier,  which  includes  a  prefix  and  a  number.  The 
prefix  is  based  on  the  type  of  data  and  the  number  represents  a  data 
element.  Table  3  illustrates  the  prefixes  used  in  each  assessment  phase. 


Assessment  Phase 

Prefixes 

Phase  1 

PR/ is  an  input. 

PRO  is  an  output. 

C  is  a  constraint. 

R  is  a  resource. 

Phase  2 

/  is  an  input. 

N  is  an  output  generated  by  one  of  Phase  2’s  activities  that 
is  not  a  final  output  of  the  phase.  (It  is  called  an  interim 
output.) 

0  is  a  final  output  of  Phase  2. 

C  is  a  constraint. 

PRO  is  an  output  of  Phase  1  that  either  acts  as  a  constraint 
or  is  used  as  a  resource  in  Phase  2. 

Phase  3 

PAI  is  an  input. 

PAO  is  an  output. 

C  is  a  constraint. 

R  is  a  resource. 

Table  3:  Dataflow  Prefixes 
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Example  Dataflow 
Identifiers 


Table  4  illustrates  the  convention  for  documenting  dataflow  identifiers 
in  MDP. 


Dataflow  Identifier 

Description 

PR02  Assessment 

scope 

The  second  output  of  Phase  1 .  It  also  acts  as  a  constraint 
for  all  Phase  2  activities. 

PR06  MDP  assess¬ 
ment  procedures 

The  sixth  output  of  Phase  1 .  It  also  acts  as  a  resource  for  all 
Phase  2  activities. 

Cl  Assessment  con¬ 
straints 

The  first  constraint  for  the  protocol.  It  can  apply  to  any 
phase  or  activity. 

12  Mission  documenta¬ 
tion 

The  second  input  of  Phase  2.  It  is  also  an  input  to  one  of 
Phase  2’s  activities. 

N2  Data  from  docu¬ 
mentation 

The  second  interim  output  of  Phase  2.  It  is  also  an  input  to 
several  of  Phase  2’s  activities. 

PAOI  Communicated 

assessment  results 

The  first  output  of  Phase  3. 

Table  4:  Dataflow  Identifier  Examples 


SOFTWARE  ENGINEERING  INSTITUTE  |  67 


References 


[Alberts  2007] 

Alberts,  Christopher,  Dorofee,  Audrey,  &  Marino,  Lisa.  Executive  Overview  of  SEI  MOSAIC: 
Managing  for  Success  using  a  Risk-Based  Approach  (CMU/SEI-2007-TN-008).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  2007. 

http://www.sei.cmu.edu/publications/documents/07.reports/07tn008.html 

[Alberts  2005] 

Alberts,  Christopher  &  Dorofee,  Audrey.  Mission  Assurance  Analysis  Protocol  (MAAP):  Assess¬ 
ing  Risk  in  Complex  Environments  (CMU/SEI-2005-TN-032,  ADA441906).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  2005. 

http://www.sei.cmu.edu/publications/documents/05.reports/05tn032.html 

[Charette  1990] 

Charette,  Robert  N.  Application  Strategies  for  Risk  Analysis.  New  York,  NY:  McGraw-Hill  Book 
Company,  1990. 

[Dorofee  1996] 

Dorofee,  A.,  Walker,  J.,  Alberts,  C.,  Higuera,  R.,  Murphy,  R.,  &  Williams,  R.  Continuous  Risk 
Management  Guidebook.  Pittsburgh,  PA,  Software  Engineering  Institute,  Carnegie  Mellon  Uni¬ 
versity,  1996. 

http://www.sei.cmu.edu/pubhcations/books/other-books/crm.guidebk.html 

[Gluch  1994] 

Gluch,  D.  A  Construct  for  Describing  Software  Development  Risks  (CMU/SEI-94-TR-014, 
ADA284922).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  1994. 
http://www.sei.cmu.edu/publications/documents/94.reports/94.tr.014.html 

[Kloman  1990] 

Kloman,  H.  F.  “Risk  Management  Agonists.”  Risk  Analysis  10,  2  (June  1990):  201-205. 

[Young  2001] 

Young,  Peter  C.  &  Tippins,  Steven  C.  Managing  Business  Risk.  New  York,  NY:  American  Man¬ 
agement  Association,  2001. 


68  |  CMU/SEI-2008-TR-005 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  search¬ 
ing  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments 
regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Head¬ 
quarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the 
Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES 


(Leave  Blank) 


March  2008 


4.  TITLE  AND  SUBTITLE 

Mission  Diagnostic  Protocol,  Version  1.0:  A  Risk-Based  Approach  for  Assessing  the  Potential 
for  Success 


6.  AUTHORfS) 

Christopher  Alberts,  Audrey  Dorofee,  and  Lisa  Marino 


7.  PERFORMING  ORGAMZATION  NAME(S)  AND  ADDRESS(ES) 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


9 .  SPONSOR!  NG'MIONTORING  AGENCY  NAIVE(S)  AND  ADDRESSES) 

HQ  ESC/XPK 
5  Eglin  Street 

Hanscom  AFB,  MA  01731-2116 


11.  SUPPLEMENTARY  NOTES 


COVERED 

Final 


5.  FUNDING  NUMBERS 

FA8721-05-C-0003 


PERFORMING  ORGAMZATION 
REPOFTT  NUMBER 

CMU/SEI-2008-TR-005 


10.  SPONSORING'MIOMTOHNG 
AGENCY  REPORT  NUMBER 


12A  DISTRIBUTION  AVAILABILITY  STATEMENT  12B  DISTRIBUnON  CODE 

Unclassified/Unlimited,  DTIC,  NTIS 


13.  ABSTRACT  (MAXIMUM 200  WORDS) 

SEI  Mission-Oriented  Success  Analysis  and  Improvement  Criteria  (MOSAIC)  is  a  managementapproach  for  establishing  and  maintain¬ 
ing  confidence  that  objectives  will  be  achieved  successfully.  It  comprises  a  suite  of  risk-based  methods  for  assessing  and  managing 
complex  projects  and  processes.  The  Mission  Diagnostic  Protocol  (MDP)  is  one  of  the  assessments  included  in  MOSAIC.  MDP  pro¬ 
vides  a  time-efficient  means  of  analyzing  the  potential  for  success  in  complex  and  uncertain  environments  and  can  be  applied  across 
the  lifecycle  and  throughout  the  supply  chain.  It  produces  a  broad  overview  of  the  current  state  of  risk  and  opportunity  for  a  projector 
process.  With  MDP,  a  set  of  key  drivers  is  evaluated  to  establish  current  conditions  and  circumstances  that  can  affect  performance. 
Then,  a  simple  algorithm  is  used  to  estimate  the  likelihood  ofachieving  the  objectives  being  pursued.  An  MDP  assessment  is  straight¬ 
forward  to  conduct,  and  it  can  be  self  applied  by  people  who  are  responsible  foroverseeing  projects  and  processes.  The  purpose  of 
this  document  is  to  define  the  framework,  orcore  set  of  activities  and  outputs,  that  define  MDP. 


14.  SUBJECT TERM6  15.  NUMBER  OF  PAGES 

Risk,  risk  management  complex  systems,  risk  analysis  tools,  risk  analysis  techniques,  risk  83 

management  tools,  risk  managementtechniques,  managing  complexity,  complex  systems 
management,  risk  analysis 


17.  SECURITY  CLASSIFICATION  OF 
REPORT 

Unclassified 


18.  SECURITY  CLASSIFICATION 

19.  SECURITY 

OF  THIS  PAGE 

CLASSIFICATION  OF 

Unclassified 

ABSTRACT 

Unclassified 

20.  UNITAT10N  OF 
ABSTRACT 

UL 


NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89)  Prescribed  by  ANSI  Std.  Z39-18 
298-102 


